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ABSTRACT 


This  paper  reports  the  test  and  evaluation  results  of  the  Integrated  Damage 
Control  Training  Technology  (IDCTT)  Trainer.  This  device  —  the  product  of  a 
tour  sear  advanced  development  effort  —  uses  interactive  courseware  which 
incorporates  the  latest  multi  media  computer  *echnology  to  create  a  realistic  damage 
control  training  environment.  The  trai:  .  •  .s  developed  to  support  a  recent 
change  in  shipboard  damage  control  philosophy  called  Total  Ship  Survivability 
(TSS);  a  concept  which  emphasizes  the  simultaneous  repairing  of  a  rhip’s  combat 
damage  while  maintaining  its  ability  to  fight. 

The  new  trainer  was  comprehensively  evaluated  using  performance  data  and 
survey  results  collected  from  students  and  instructors  during  a  three  month  test 
period  at  the  Surface  Warfare  Officer  School’s  Damage  Control  School  in  Newport, 
R.I.  Findings  from  seven  different  surveys  are  presented;  performance  comparisons 
between  the  conventional  trainer  and  this  new  trainer  are  examined;  and  narrative 
accounts  of  both  students  and  instructors  are  reported.  The  data  clearly  identify 
and  isolate  the  specific  benefits  as  well  as  some  drawbacks  associated  with  the 
various  enabling  technologies  integrated  during  the  advanced  development  of  the 
prototype.  Recommendations  about  operationally  deploying  the  device  are  discussed 
and  the  implications  of  suggested  enhancements  are  explored. 
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EXECUTIVE  SUMMARY 


Recent  developments  in  multimedia  technology  have  made  it  possible  for 
computer  generated  simulations  to  create  vivid,  life-like  damage  control  training 
scenarios.  The  Navy's  damage  control  community  —  acquisition  agents,  trainers, 
and  research  and  development  managers  —  has  explored  these  new  technologies  to 
produce  more  realistic  training  scenarios.  One  such  exploration  culminated  in 
a  multimedia  integration  of  several  enabling  technologies  collectively  called 
IDCTT:  the  Integrated  Damage  Control  Training  Technology. 

The  Integrated  Damage  Control  Training  Technology  (IDCTT)  Trainer  is  an 
interactive  video  courseware  (ICW)  medium  which  incorporates  the  latest  video  and 
audio  technology  to  provide  realistic  damage  control  training  in  a  simulated 
shipboard  environment.  IDCTT' s  design  goal  was  to  provide  Damage  Control 
Assistants  (DCAs)  with  interactive  computer-based  damage  control  scenarios  that 
realistically  showed  the  consequences  of  their  damage  control  decisions  in  terms 
of  their  impact  on  the  shipboard  environment.  This  thesis  evaluated  the  extent 
to  which  this  interactive  video  courseware  achieved  that  design  goal. 
Specifically,  it  validated  the  effectiveness  of  the  IDCTT  Trainer  as  a  training 
tool  for  Damage  Control  Assistants  (DCAs)  and  the  instructors  who  train  them. 

IDCTT  was  specifically  designed  to  support  DCA  training,  not  in  the 
conventional  sense,  but  under  relatively  recent  changes  in  shipboard  damage 
control  philosophy.  The  results  of  this  doctrinal  shift  has  been  called  the 
Total  Ship  Survivability  (TSS)  concept.  Total  Ship  Survivability  reasserts  the 
World  War  II  concept  of  simultaneously  repairing  combat  damage  while  maintaining 
the  ship's  ability  to  fight.  TSS  is  a  training  concept  designed  to  increase  a 
ship's  war  fighting  capability  and  the  IDCTT  Trainer,  in  turn,  is  designed  to 
deliver  that  TSS  based  training  concept. 

This  paper  examined  two  central  aspects  of  the  IDCTT  Trainer's  performance: 
its  performance  in  and  of  itself,  and  its  performance  when  compared  to  the 
trainer  currently  in  use,  which  is  called  the  Damage  Control  Central  Trainer. 
Pinpointing  these  relatively  circumscribed  areas  enabled  the  evaluation  and  the 
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subsequent  analyses  to  produce  sufficient  data  needed  to  quantify  the  impact  the 
new  integrated  technologies  had  upon  both  student  and  instructor  performance. 

The  test  and  evaluation  data  collection  plan  for  the  device  was  designed 
to  solicit  and  collect  student  and  instructor  inputs,  which  taken  together, 
provided  a  basis  upon  which  to  evaluate  how  well  IDCTT  supported  DCA  training. 
This  portion  of  the  study  identified  system  strengths,  weaknesses,  and  design 
features  of  the  prototype  which  could  be  improved  to  enhance  its  next  variant 
and  subsequent  generations  of  the  product.  Also,  the  measures  of  student  and 
instructor  reactions  to  IDCTT  provided  a  basis  from  which  to  gain  insights  into 
the  usefulness  of  the  trainer  as  a  fleet  training  aid.  Students  and  instructors 
provided  this  information  using  short  essays  and  descriptive  narrative  accounts, 
quantitative  subjective  rating  data,  check-off  lists  of  problem  features,  formal 
surveys,  and  both  structured  and  unstructured  interviews. 

A  direct  comparison  of  IDCTT  with  the  DC  Central  Trainer  was  done  to 
concurrently  determine  the  relative  effectiveness  of  both  trainers.  The 
evaluation  sought  to  draw  comparisons  by  obtaining  student  performance  scores  for 
each  trainer  type  on  a  variety  of  comparative  dimensions,  and  on  measures  of  each 
systems'  ability  to  produce  standardized  training  scenarios.  This  information 
was  obtained  during  student  training  sessions  conducted  in  both  trainers  during 
which  performance  was  carefully  graded  using  a  standardized  grading  protocol. 
Moreover,  subjective  impressions  of  user  preferences  were  quantified  using 
standardized  rating  scales  designed  to  further  compare  and  contrast  design 
features  of  both  trainers. 

IDCTT 's  test  and  evaluation  was  conducted  at  the  Damage  Control  Training 
Department  of  the  Surface  Warfare  Officer  School  in  Newport,  Rhode  Island.  This 
school  provides  the  only  academic  training  for  novice  DCAs .  Before  this  study, 
the  School's  only  source  of  simulated  DCA  battle  problem  training  was  the 
conventional  Damage  Control  Central  Trainer.  The  new  IDCTT  Trainer  was 
transported  to  the  DCA  School  and  installed  in  its  simulation  wing  for  the 
purpose  of  the  current  test  and  evaluation  study. 

Data  were  collected  on  three  occasions.  The  first  test  period,  24  through 
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29  March  1994,  consisted  of  preliminary  trials  designed  to  validate  the  planned 
data  collection  methodology  and  to  evaluate  the  IDCTT  Trainer's  performance 
characteristics.  Findings  from  this  pilot  test  phase  were  used  to  modify  the 
original  test  and  evaluation  data  collection  plan  as  well  as  the  actual  hardware 
configuration  of  the  IDCTT  trainer  prototype  itself.  The  validation  data  were 
collected  during  the  second  and  third  test  periods  which  were  conducted  19 
through  21  April  1994,  and  20  through  22  June  1994.  Thirty-two  students  and 
seven  instructors  participated  in  the  evaluation. 

The  validation  data  revealed  that  the  IDCTT  Trainer  was  highly  effective 
on  the  majority  of  *  lining  dimensions  specifically  evaluated  by  the  test 
methodology.  For  example,  students  reported  that  the  IDCTT  Trainer  was  easy  to 
operate  and  extremely  useful  as  a  training  aid  at  the  Damage  Control  School. 
They  also  reported  their  desire  to  see  the  trainer  made  available  for  shipboard 
use.  When  the  IDCTT  Trainer  was  directly  compared  to  the  DC  Central  Trainer, 
students  reported  that  the  IDCTT  Trainer  was  clearly  the  preferred  training 
method.  IDCTT  promoted  a  more  rapid  acquisition  of  basic  skills  and  ultimately, 
a  higher  level  of  proficiency.  Moreover,  contrasted  to  the  conventional  DC 
Central  Trainer,  IDCTT  induced  significantly  higher  levels  of  stress  and 
motivated  students  to  more  actively  participate  in  the  damage  control  scenario. 
Formal  statistical  tests  of  the  differences  between  scores  from  the  two  different 
trainers  revealed  that  the  IDCTT  scores  were  in  fact  higher  than  the  conventional 
DC  Trainer's  (p  <  .05). 

Although  students  and  instructors  reported  that  the  IDCTT  Trainer  was  an 
effective  training  medium,  they  did  identify  configural  and  functional 
characteristics  of  the  trainer  for  potential  improvement.  For  example,  some  of 
these  candidate  changes  were  associated  with  using  the  touchscreen  monitor  to 
input  information  into  the  system.  Fifty-nine  percent  of  the  students  reported 
that  they  experienced  some  difficulty  operating  this  input  device.  The  screen's 
primary  problem  was  its  low  sensitivity  to  touch  and  its  slow  response  time 
Students  recommended  increasing  its  sensitivity  or  simply  exploring  different 
input  methods,  such  as  voice  activation  for  future  consideration.  The 
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touchscreen  design  feature  was  but  one  of  several  identified  by  this  evaluation. 
The  present  report  lists  other  suggested  modifications  to  the  system  and  offers 
substantiation  to  support  the  need  for  reconfiguration  in  future  variants. 

The  collective  reports  from  both  student  officers  and  instructors  who  used 
IDCTT's  technologies  revealed  one  broad,  simple  finding:  the  trainer  was  as  well 
liked  by  its  users  as  it  was  effective,  especially  when  compared  to  its 
conventional  counterpart.  As  good  as  IDCTT  was  however,  students  and  instructors 
freely  suggested  ways  to  make  it  better.  The  findings  and  recommendations 
contained  in  this  report,  therefore,  highlight  the  indispensable  role  formal 
test  and  evaluation  methodologies  play  in  the  transition  of  educational  products 
from  advanced  development  to  service  use. 


I .  INTRODUCTION 


Throughout  American  Naval  History,  damage  control  has  played  a  key 
role  in  the  success  of  American  warships  at  sea.  The  fundamental 
requirement  needed  to  prepare  for  battle  damage  is  damage  control  training 
and  since  it  is  unreasonable  for  ships  to  actually  incur  self-imposed 
damage,  damage  control  training  has  come  to  employ  simulations  of  battle 
damage  situations.  In  the  past,  these  scenarios  were  relatively  crude, 
re.ymg  heavily  upon  human  intensive  role  playing.  Recent  developments  in 
multimedia  capabilities  however  have  made  it  possible  for  computer 
generated  simulations  to  create  life-like  training  scenarios.  The  damage 
control  community  has  adopted  these  new  technologies  to  produce  more 
realistic  training  scenarios.  One  such  adaptation  is  a  multimedia  package 
called  the  Integrated  Damage  Control  Training  Technology  (IDCTT). 

The  Integrated  Damage  Control  Training  Technology  (IDCTT)  Trainer  is 
an  interactive  video  courseware  (ICW)  medium  which  incorporates  the  latest 
video  and  audio  technology  to  provide  damage  control  training  in  a 
simulated  shipboard  environment.  This  thesis  addresses  that  interactive 
video  courseware.  Specifically,  it  validates  the  effectiveness  of  IDCTT 
as  a  training  tool  for  Damage  Control  Assistants  (DCAs)  and  its  results 
will  be  used  by  the  Naval  Personnel  Research  and  Development  Center 
iNPRDCi  to  justify  the  use  of  the  IDCTT  system  as  a  fleet  training  aid. 

The  focus  of  the  present  chapter  is  threefold.  First,  background 
information  concerning  the  IDCTT  Trainer  and  the  fundamental  concepts  of 
damage  control  are  discussed.  Second,  a  detailed  description  of  the  new 
multimedia  components  used  to  create  the  Interactive  Damage  Control 
Training  Technologies  (IDCTT)  Trainer  is  provided.  A  description  of  the 
user's  interaction  in  this  new  medium  is  included  in  the  discussion. 
Finally,  the  conventional  shore  based  training  technique  in  use  today,  the 
Damage  Control  Central  Trainer,  is  summarized. 
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BACKGROUND 


::  .'TT  ■•■'as  sp«r  if:  :al  l4..  iesijned  to  support  DC  A  training,  not  in  the 
r.  r.a .  sense,  rut  ur.de:  a  relatively  recent  change  in  shipboard 
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.•ir.ult  ar.ecusly  repairing  combat  damage  while  maintaining  the  ship's 
ability  to  tight.  Afte*  the  ’JSS  Stark  198“  and  USS  Samuel  B.  Roberts 
1 v S S  >  incidents,  it  became  apparent  that  the  deunage  control  organization 
was  able  tc  repair  damage  necessary  to  save  the  ship  but  in  each  incident, 
the  ship  lost  the  ability  to  defend  itself  from  further  attacks.  TSS  is 
a  training  concept  designed  to  increase  a  ship's  war  fighting  capability 
and  the  IDCTT  Trainer  is  designed  to  deliver  that  TSS  based  training 
concept . 

The  IDCTT  Trainer  uses  information  from  the  Battle  Damage  Estimator 
(BDE)  which  is  a  software  package  that  displays  the  most  probable  damage 
to  a  ship  after  being  hit  with  a  particular  weapon.  Moreover,  IDCTT 
interfaces  with  the  Integrated  Survivability  Management  System  (ISMS),1 
which  provides  two  dimensional  graphic  representations  of  the  ship,  to 
produce  line  diagram  drawings  of  damaged  areas  and  systems.2  It  also  uses 
the  latest  laser  disk  video,  compact  disk  audio,  and  computer  technology 
to  create  a  realistic  training  environment. 

1 .  Damage  Control  Fundamentals 

A  basic  understanding  of  what  Naval  Damage  Control  is  and  how  it 
is  conducted  is  needed  to  understand  what  IDCTT  attempts  to  accomplish. 
Damage  Control  is  divided  into  two  disciplines;  damage  prevention,  and 
damage  containment  and  repairs.  The  IDCTT  Trainer  focuses  on  damage 


;  The  ISMS  system  was  removed  from  the  IDCTT  Trainer  after  initial  test 
results  revealed  human  interface  problems  which  will  be  discussed  later  in  the 
text . 


'  The  Battle  Damage  Estimator  (BDE)  and  the  Integrated  Survivability 
Management  System  (ISMS)  will  be  described  in  detail  later  in  the  text. 
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r."  a ;  r.r  ar.  i  :«pa:rs  This  aspact  f  damag*  centre!  i»  dafinad  as 

i  \r,.i  1  - *a . : ze  1amag«  -.-her.  .*  incurs  by  measures  such 

.  ...r.j  :  . :  .-air.  3  p:  es«:  v;r.  g  s  *.  at  1 ! ;  r  •.  and  buoyancy  . 

.  .:.u  i  -  if.*  ir.g  r-pla  ir.c  assent  :a  1  structures  and 

c  essur.*  la  »  eq-ipmer.r  Jr  it  sen  j  1  >  =  . 

2.  Damage  Control  Organization 

Tmce  this  tr.es  is  repeatedly  refers  to  members  cf  the  damage 
.  ;  r.t  r  *  .  ;  :  jar.  1  cat  1  rr.  this  crgar.i  cat  icn  will  be  described  in  some  detail. 
::  11.  ir.c  provides  'he  reader  with  a  general  discussion  of  the 

:pei  at  1  rr.a  .  damage  centre,  organization  . 

a.  Damage  Control  Aaaiatant 

The  Damage  Control  Assistant  DCA>  reports  to  the  Chief 
Engineer  who  is  the  Damage  Control  Officer  and  responsible  for  deunage 
control  or.  the  ship.  He  also  reports  to  the  Officer  of  the  Deck-’  who  is 
the  Commanding  Officer's  direct  representative  on  all  operational  matters. 
The  DC A  uses  repair  lockers  II,  III,  and  V,4  which  are  equipped  with  the 
appropriate  damage  control  equipment  and  manned  with  20  to  50  personnel, 
to  combat  shipboard  damage.  The  Combat  Systems  Maintenance  Center  (CSMC) 
provides  an  interface  between  the  DCA  and  Combat  Systems  personnel  to 

coordinate  the  alignment*  and  repairs  of  the  ship's  weapon  systems. 

b.  Damage  Control  Central 

Damage  Control  Central  (DC  Central)  is  a  command  center  from 
which  the  DCA  coordinates  shipboard  deunage  control  efforts.  A  support 
organization  in  DC  Central  assists  the  DCA  in  collecting,  processing. 


The  Officer  of  the  Deck  stands  watch  on  the  bridge  and  is  charged  with 
coordinating  shipboard  operations  and  given  the  authority  to  make  operational 
decisions  in  the  Commanding  Officer's  absence.  He  is  also  required  to  keep  the 
Commanding  Officer  informed  of  all  pertinent  shipboard  matters. 

4  Repair  II  and  III  are  responsible  for  damage  control  in  the  forward  and 
after  areas  of  the  ship,  respectively.  Repair  V  is  charged  with  damage  control 
in  the  engineering  propulsion  spaces. 

*  Due  to  redundancies  built  into  ships'  weapon  systems,  there  are  various 
component  configurations  that  can  be  used  to  make  a  system  operational. 
Alignment  of  weapon  systems  refers  to  the  selection  of  available  components  to 
make  a  weapon  system  operational. 
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>r.i . .  o.r.j  ir.  i  i.ssen;nat  ;ng  damage  control  information.  This  support 
: .  ..  ies  j;uni  Powered  Phone  Talkers,  Plotters  and  the  Damage 

•  '  :  r.  t  e:  at  ;  :  .  These  toes  are  described  below. 

Power ea  Phone  Talker.  The  Sound  Powered  Phone 

'•  ■  r-.-  ,.-ir.c  s'ar. iard  phraseology  between  the  DCA  and 

-•  •  >*  .  r.-.rir  st  at  i  _r.s  include  Repair  Lockers  II,  III  and  V, 


Li;:.  .'SMC .  and  the  Bridge. 

C  Damage  Control  Plotter.  The  Damage  Control  Plotter 

clots  all  emergencies  and  damage  control  actions  taken  using  standard 
damage  control  symbology  on  the  Damage  Control  Plates.6  He  also  assist 
the  DCA  by  identifying  and  recommending  damage  control  actions  based  on  an 

analysis  of  the  plot  on  the  Damage  Control  Plates. 

■j  Damage  Control  Console  Operator.  The  Deunage  Control 

Console  Operator  (DCCO)  monitors  an  alarm  display  panel  which  remotely 
indicates  damaged  systems  and  compartments.  The  DCCO  also  monitors  the 
Firemain  Alarm  Panel  and  Pump  Logic  Diagram.  This  system  reports  firemain 
pressure,  indicates  firemain  valve  positions,  and  displays  which  fire 
pumps  are  operating.  The  DCCO  can  open  or  shut  firemain  valves  and  start 
or  stop  fire  pumps  from  this  console. 

B.  The  IDCTT  System 

The  IDCTT  Trainer  is  the  result  of  research  conducted  to  fulfill  fleet 
requirements  for  more  realistic  training  in  both  the  shipboard  and 
training  command  environments. 

The  objective  of  IDCTT  is  to  provide  interactive  computer-based 
damage  control  scenarios  to  Damage  Control  Officer  and  enlisted 
students .  IDCTT  scenarios  provide  an  interactive  decision 

environment  that  show  ship  damage  control  decision  consequences 
(Ulozas,  p.l,  15  November  1993). 

This  technology  exploits  the  ability  of  computer  based  training  to 
simulate  scenarios  that  cannot  be  feasibly  replicated  in  the  real  world 


’  Damage  Control  Plates  are  blueprints  of  the  ship  that  the  DCA  uses  to 
track  initial  damage  and  the  progression  of  fires  and  flooding.  Included  in 
these  plates  are  piping  overlays  which  display  the  network  of  firemain,  fuel, 
chill  -water,  ventilation,  and  compressed  air  pipes  and  ducts. 
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for  training  purposes.  IDCTT  attempts  t.o  realistically  duplicate  stressful 

renditions  that  a  DCA  would  encounter  under  actual  battle  conditions. 

Long  standing  research  on  decision  making  under  stress  suggests  that 
these  untrained  and  unprepared  to  act  in  these  crisis  environments 
tend  to  make  poorer  decisions  than  those  traired  repeatedly  in  quasi- 
realistic  scenarios.  At  present,  such  training  does  not  exist  in  the 
formal  school  context.  (Ulozas,  p.l,  15  November  93) 

The  IDCTT  Trainer  manipulates  the  stress  level  imposed  on  the 
student  DCA  by  addressing  six  different  stimulus  features  listed  below. 

•  The  volume  of  information  conveyed, 

•  The  rapidity  with  which  information  is  conveyed  from  various  sources, 

•  The  extent  to  which  stimuli  are  partially  masked  by  extraneous 
ambient  noise, 

•  The  presence  of  distractors  such  as  flashing  video  and  loud  audio 
alarms , 

•  The  onset  of  unexpected  status  report  inquiries  from  the  ship's 
Commanding  Officer,  and 

•  Negative  feedback  from  superiors  for  inappropriate  decisions. 

Since  the  combined  effort  of  the  ship's  crew  is  required  to  fight  any 
damage  control  problem  'damage  control  problems  are  generally  exercised 
while  the  ship  is  at  General  Quarters  (GQ)),~  training  time  for  DCAs  is 
limited.  IDCTT  enables  the  DCA  to  realistically  exercise  decision  making 
skills  in  a  worse  case  scenario,  without  effecting  the  ship's  routine. 
Moreover,  it  provides  shore  commands  with  a  more  realistic  training 
environment . 

The  IDCTT  depends  on  inputs  from  autonomous  systems  to  create 
realistic  training  scenarios  and  uses  the  most  modern  technology  to 
display  these  scenarios  to  the  user.  These  system  inputs,  hardware 
requirements,  and  methods  used  to  implement  them  in  the  IDCTT  Trainer  are 
discussed  in  the  following  sections.  The  first  section,  " IDCTT  System 
Inputs",  describes  how  the  Battle  Damage  Estimator  (BDE)  and  Integrated 
Survivability  Management  System  (ISMS)  interface  with  the  IDCTT  Trainer. 


General  Quarters  is  the  highest  condition  of  shipboard  readiness  in  which 
the  entire  crew’  mans  their  respective  battle  stations. 
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The  second  section,  " IDCTT  Trainer  Hardware ",  describes  each  of  the 
components  used  by  IDCTT  tc  create  a  realistic  training  environment.  The 
third  section,  "IDCTT  Trainer  Software" ,  outlines  the  computer  program 
used  to  run  the  IDCTT  Trainer.  Finally,  the  section,  ‘IDCTT  Trainer 
Tt  i  i  i  zat  i  or.  “ ,  describes  how  the  system  is  manned  and  operated  and  a  brief 

'verviev  ::  the  battle  problem  scenario  is  given. 

1  .  IDCTT  SYSTEM  INPUTS 

The  BDE  and  ISMS  are  independent  systems  which  provide  inputs  to 
the  IDCTT.  The  IDCTT  Trainer  uses  these  inputs  to  provide  outputs  in  the 
form,  of  user  training  scenarios.  The  BDE  provides  the  information 
necessary  to  compose  accurate  training  scenarios  that  reflect  the  most 
probable  damage  for  various  weapon  hits.  The  ISMS,  although  an  independent 
system,  interfaces  with  the  IDCTT  hardware  to  provide  various  information 
directly  tc  the  system.  These  two  subsystems  are  described  more  fully 
below . 

a.  Battle  Damage  Emtimator (BDE) 

The  BDE  was  developed  on  a  personal  computer  to  display 

probable  shipboard  battle  damage,  based  on  a  model  called  the  Ship 

Vulnerability  Model  (SVM) ,e  The  BDE  enables  a  user  to  choose  a  weapon  from 

a  menu  of  eight  types,"  and  an  impact  point  on  the  ship.  Using  these  two 

inputs,  the  BDE  then  provides  damage  estimates  for  the  specific  weapon 

chosen  at  the  specified  hit  location.  This  estimate  includes  a  three 

dimensional  image  indicating  all  spaces  that  would  be  flooded  or  on  fire. 

Further  damage  information  is  provided  in  seven  areas:  (1)  hull, 
mechanical,  electrical  and  combat  systems,  (2)  crew,  (3)  firemains, 
i4)  chill  water  mains,  (5)  electrical  power  panels  and  cable  runs, 

! 6:  high  pressure  air  mains,  and  (7)  low  pressure  air  mains  (David 


The  Ship  Vulnerability  Model  was  developed  by  the  David  Taylor  Research 
Center  with  support  from  Naval  Sea  Systems  Command  to  provide  extremely  detailed 
models  of  the  probable  damage  caused  by  different  conventional  weapons  on  a 
combatant  ship.  This  model  is  based  on  actual  weapon  hit  data  from  the  Gulf  War 
and  shock  test  data  obtained  from  various  types  of  weapons. 

'  Weapon  types  include  Exocet,  Harpoon,  Stinger,  cruise  and  surface  to  air 
missiles,  contact  and  influence  mines  as  well  as  projectiles. 
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Taylor  Research  Center  Ship  Structures  and  Protection  Department, 
p . 2 ,  June  1992  )  . 

Information  generated  by  the  BDE  is  used  to  develop  training 
scenarios  which  are  then  used  by  the  IDCTT  Trainer.  Using  the  BDE  allows 
scenarios  to  be  developed  from  a  standardized  data  base  of  probable  ship 
damage,  thus  eliminating  the  developers'  need  to  approximate  what  damage 
might  occur  based  on  personal  opinion.  BDE  output  used  to  develop  training 
scenarios  has  been  successfully  demonstrated  by  the  Afloat  Training 
Organization  (ATO!  TSS  drills  during  Refresher  Training  (REFTRA)  since 
-9S“.  The  ATOs  use  the  Total  Ship  Survivability/Fleet  Training  Model, 
a  model  similar  to  the  BDE  and  also  based  on  the  SVM,  to  provide  damage 

estimates  for  conventional  weapon  hit  scenarios. 

b.  Integrated  Survivability  Management  Systems  (ISMS) 

ISMS  is  a  NAVSEA  project  designed  to  meet  the  information, 
communication,  and  command  and  control  needs  imposed  on  DCAs  by  the  added 
complexity  of  state  of  the  art  ships  and  TSS  responsibilities.  The  goal 
of  ISMS  is  to  provide  more  information  to  the  DCA  in  a  clear  compact  form. 
Specifically  ISMS  helps  the  DCA  by: 


•  Determining  the  type  and  location  of  weapon  effects, 

•  Communicating  this  information  to  decision  stations, 

•  Displaying  the  information, 

•  Integrating  the  information  with  the  ongoing  activities, 

•  Developing  plans  of  action, 

•  Initiating  commands,  and 

•  Executing  the  commands  by  merging  traditional,  survivability  efforts 
with  improved  communications  and  computer  support  (Naval  Sea  System 
Command,  p.l,  1992). 


The  ISMS,  which  is  run  on  a  Sun  SPARC  10  computer,  was  integrated 
into  the  IDCTT  system  to  provide  students  with  all  DC  plate  and  piping 
overlay  information.  Prior  to  the  ISMS  system,  this  information  was 


The  Afloat  Training  Organization  (ATO)  is  a  shore  installation  which 
embodies  officer  and  enlisted  training  experts.  The  ATO  is  responsible  for 
training  ships  in  various  mission  areas  and  quantitatively  grading  their 
performance  during  Refresher  Training  (REFTRA).  REFTRA  is  a  periodic  18  month 
training  requirement  where  ships  undergo  intensive  fundamental  training  on  all 
aspects  cf  surface  warfare. 
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depicted  cn  laminated  blueprints  to  which  the  DC  A  would  refer  for 
information.  ISMS  now  enables  students  to  zoom  in  on  any  portion  of  the 
ship  enabling  them  to  obtain  exact  compartment,  access  and  valve  number 
information.  IDCTT  also  updates  the  computer  generated  damage  control 
plates  with  red  and  blue  shadings  to  clearly  indicate  fire  and  flooding, 
respectively.  Finally,  ISMS  uses  damage  reports  from  the  user's  inputs  to 
the  IDCTT  to  update  damage  control  efforts  in  each  affected  space  using 

standard  damage  control  symbology. 

2  .  IDCTT  Trainer  Hardware 

The  IDCTT  Trainer  integrates  the  latest  multimedia  hardware  to 
produce  damage  control  scenarios  that  simulate  the  shipboard  conditions  a 
DCA  would  actually  face  in  the  event  of  shipboard  damage.  The  system 
combines  a  personal  computer,  monitors,  laser  disk,  printer,  CD  rom, 
speakers,  and  a  Sun  SPARC  10  workstation  to  fully  emerse  the  user  in  a 
realistic  simulated  shipboard  environment.  These  components  and  their 
application  are  described  below. 

a.  IBM  Compatible  Personal  Computer 

A  486  IBM  compatible  personal  computer  is  the  hardware  basis 
of  the  IDCTT  system.  This  computer  provides  the  scenario  event  time  line 

and  updates  the  training  scenario  based  on  student  inputs. 

Jb.  Computer  Monitors 

The  system  uses  a  21  inch  touchscreen  monitor  and  two  15  inch 
monitors.  The  21  inch  touchscreen  monitor  is  the  input  device  for  student 
orders  pertaining  to  specific  damage  control  actions.  These  orders  are 
displayed  in  a  menu  of  damage  control  options  used  by  the  student  to 
combat  the  damage.  The  monitor's  upper  left  portion  depicts  reports  from 
various  DC  Central  watchstanders  that  the  DCA  receives  throughout  the 
scenario . 

The  two  remaining  15  inch  monitors  are  used  to  provide  a 
Damage  Control  Alarm  Panel  and  a  Firemain  Alarm  Panel  and  Pump  Logic 
Diagram.  The  Damage  Control  Alarm  Panel  alerts  the  user  when  space  alarms 
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have  been  activated.  The  Firemam  Alarm  Panel  indicates  system  pressure, 
firemain  value  position  (opened  or  closed),  and  fire  pump  operations  (on 

or  off'  . 

c.  Laser  Disk  Player 

The  laser  disk  player  is  used  to  provide  video  images  to  the 
touchscreen  monitor.  These  images  graphically  depict  shipboard  personnel 
providing  information  to  the  DCA  and  subsequently,  personnel  relaying 
orders  from  the  DCA  to  the  appropriate  crew  members. 

d.  Printer 

A  laser  printer  receives  preprogrammed  graphic  output  from 
tne  personal  computer  providing  the  user  with  Damage'  Control  Chits.11 
These  chits  are  printed  out  and  available  for  the  DCA  to  review  as  a 

backup  tc  the  voice  reports  received  over  the  Command  and  Control  Monitor. 

e .  CD  Rom 

The  CD  rom  provides  all  scenario  audio  inputs  not  included 

m  the  laser  disk  output  such  as  alarm  sirens,  and  background  noise. 

f .  Speakers 

The  speakers  used  in  the  IDCTT  system  provide  stereo  sound 
from  two  separate  input  sources.  The  laser  disk  player  provides  scenario 
audio  inputs  while  the  CD  Rom  simultaneously  blends  background  noise  to 

create  a  realistic  audio  facsimile  of  a  shipboard  locale. 

g.  Sun  SPARC  10  Workstation 

The  Sun  SPARC  10  Workstation  hosts  the  ISMS  system,  as.  This 
system  is  connected  to  the  personal  computer  through  a  one  way  interface 
from  the  personal  computer  to  the  ISMS  system.  The  ISMS  system's  graphical 
representations  are  updated  based  on  the  scenario  event  time  line  received 
from  the  personal  computer. 


Damage  Control  Chits  are  hand  written  notes  used  in  the  fleet  to 
communicate  between  the  DCA  and  various  personnel.  They  provide  information  on 
damage  location  and  efforts  to  contain  and  repair  the  damage. 
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IDCTT  Trainer  Software 
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The  IDCTT  scenario  is  written  in  the  object  computer  language 
“,'uest".  Tne  program  uses  a  next  event  time  advance  mechanism  called  the 
“event -scheduling  approach".'-  This  prototype  software  was  developed  by 
t  r.e  Center  for  Interactive  Media 

The  computer  program's  design  goal  was  to  develop  an  automated,  test 
question  derived  model  for  designing  comprehensive,  test  performance 
niver..  remedial  instruction  which  alternately  tests  and  instructs 
v.til  mastery  is  complete  .Surface  Warfare  Officer  School  Command 
Report,  Froclerr.  Description  and  Needs  Justification  for  Interactive 
Damage  Control  Training  Module,  p.3.  4  February  1993)  . 

4.  IDCTT  Trainer  Utilization 

The  IDCTT  Trainer  was  assembled  next  to  the  existing  DC  Central 
Trainer  at  the  Damage  Control  School .  This  provided  the  Damage  Control 
School  with  a  computer  based  training  medium  unlike  any  training  method 
previously  used  in  the  damage  control  community.  To  understand  how  the 
school  implemented  this  system,  a  familiarization  of  how  the  system  is 
manned,  operated,  and  what  the  battle  problem  scenario  entailed  is 

desirable.  These  topics  are  addressed  below. 

a .  Manning 

The  IDCTT  Trainer  was  designed  to  be  operated  by  one  or  two 
users.  When  operated  by  a  single  user,  the  user  assumes  the  role  of  a  DCA 
who  must  perform  all  plotting  responsibilities.  When  two  users  operate 


The  event  scheduling  approach  to  simulation  modeling  is  a  method  where 
future  events  are  explicitly'  coded  into  the  model  and  are  scheduled  to  occur  in 
the  simulated  future  (Simulation  Modeling  and  Analysis,  p.12,  1991).  The 
simulated  future  is  the  models  best  approximation  of  what  would  happen  in  the 
real  future,  given  a  specific  set  of  circumstances.  For  example,  if  a  valve  is 
ordered  closed,  the  program  acknowledges  the  order  and  schedules  the  valve  to  be 
closed  at  clock  time  plus  a  previously  programmed  delay  time  representing  the 
action  of  physically  closing  the  valve. 

CIM,  an  interactive  media  research  facility  located  in  Bethesda, 
Maryland,  developed  the  software  for  the  IDCTT.  This  software  development  was  in 
direct  support  of  training  requirements  established  by  the  Naval  Personnel 
Research  and  Development  Center  and  the  Naval  Sea  Systems  Command.  CIM  gained 
expertise  in  developing  interactive  software  through  its  ground  breaking  work  on 
the  Computer  Aided  Medical  Information  System  (CAMIS) ,  a  program  used  to  train 
medical  students  in  operating  room  and  triage  procedures. 
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the  IDCTT  Trainer,  one  assumes  the  role  of  DCA;  the  other  assists  as  the 
plotter . 

(1;  One  User.  When  one  user  mans  the  system,  that  person 
monitors  ail  alarm  panels,  voice  reports,  and  printed  DC  Chits. 
Simultaneously,  the  user  must  input  orders  to  the  Command  and  Control 
Console  and  locate  damage  control  information  from  the  ISMS  or  DC  Plates. 
The  most  time  consuming  action  for  the  single  user  is  locating  valve, 
bulkhead,  fitting,  and  compartment  numbers  from  the  ISMS  or  DC  Plates, 

actions  normally  done  by  a  Plotter. 

(2!  Two  Users.  With  two  users  assigned,  there  is  a 

convenient  division  of  duties:  one  user  takes  command  as  DCA;  the  other 
assists  as  a  Plotter,  providing  the  appropriate  information  from  the  ISMS 
or  the  DC  Plates.  For  this  study,  teams  of  two  persons  were  assigned  to 

the  trainer.  This  will  be  discussed  more  fully  in  the  methods  section. 

b.  Operation 

The  program  is  initiated  using  a  start  option  on  the  Command 
Console  menu  screen.  The  DCA  has  the  option  to  select  essential 
watchstanders  and  stations  from  the  Command  Console  menu  from  which  he  can 
request  information  or  direct  action.  Once  a  watchstander  or  station  is 
selected,  a  second  menu  replaces  the  previous  one  with  a  list  of  options 
that  the  watchstander  or  station  can  perform.  This  process  can  be 
repeated  until  the  desired  action  a  student  wishes  to  order  is  displayed 
on  the  menu.  When  an  action  is  chosen,  the  computer  generated  audio 
visual  representation  of  the  appropriate  DC  Central  watchstander  back  the 
information  the  DCA  has  ordered  and  relays  it  to  the  appropriate  station. 

All  student  inputs  to  the  system  (orders)  affect  the 
simulation  event  time  line.  When  an  order  is  given,  the  required  action 
is  scheduled  into  the  future.  Some  events,  such  as  the  mine  and  missile 
hits,  are  programmed  to  occur  regardless  of  the  user's  actions.  This 


process  provides  each  student  with  the  same  baseline  scenario,  but 


significant  variation  in  the  scenario  is  induced  depending  on  the  user's 

act  ions . 

c.  IDCTT  Scenario 

The  IDCTT  scenario  starts  with  the  ship  going  to  General 
Quarters.  During  this  evolution,  material  condition  Zebra:<  is  set 
throughout  the  ship  which  the  DCA  monitors  and  reports  to  the  Bridge. 
After  condition  Zebra  is  set,,  the  ship  takes  a  mine  hit  aft  of  the  after 
Vertical  Launch  System  (VLS)  Magazine.15  The  mine  hit  produces  fires 
adjacent  to  the  aft  VLS  Magazine,  flooding  below  the  waterline,  and  shock 
damage  to  various  systems.  Depending  on  the  actions  taken  by  the  DCA,  the 
damage  may  spread,  be  contained,  or  repaired. 

After  the  DCA  has  had  approximately  seven  minutes  to  respond 
tc  the  mine  damage,  an  Exocet  missile  impacts  the  ship  forward  of  CIC  on 
the  0-1  level.-0  The  missile  inflicts  a  rupture  to  the  chill  water  system 
that  renders  the  radar  system  inoperative  and  causes  various  fires  in  the 
vicinity  of  the  blast.  Similar  to  the  mine  hit,  damage  may  spread,  be 

contained,  or  repaired  depending  on  the  DCA's  actions. 

d.  Kill  Points  and  Pitfalls 

The  IDCTT  Trainer  scenario  uses  "kill  points"  to  terminate 
the  training  evolution  when  it  is  determined  by  the  software  that  the 
student's  actions  would  result  in  a  complete  loss  of  the  ship.  For 
incorrect  actions  with  less  severe  consequences,  the  program  uses 


'4  Zebra  is  the  code  name  used  to  describe  the  highest  material  readiness 
condition  a  ship  can  maintain.  In  condition  Zebra,  all  water-tight  doors  and 
hatches  are  closed  to  prevent  progressive  fire  spread  and  flooding  in  the  event 
of  damage  Further,  key  valves  are  closed  in  piping  systems,  thus  breaking  them 
into  smaller  systems,  to  prevent  damage  in  one  area  from  effecting  the  entire 
svstem . 


The  VLS  Magazine  contains  the  ship’s  anti-air  and  anti-submarine  missile 
inventory.  There  is  one  VLS  Magazine  located  in  the  forward  and  after  areas  of 
the  ship. 

C-l  level  indicates  the  first  deck  above  the  main  deck.  Continuing  this 
standard  numbering  system,  the  0-2  level  is  the  second  deck  above  the  main  deck. 
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"pitfalls'1  to  warn  the  student  that  an  inappropriate  action  was  taken. 

"Kill  points"  and  "pitfalls"  are  described  below. 

i  :  1  Kill  Points.  The  IDCTT  Trainer  has  five  distinct  kill 

points  which  will  terminate  the  program.  Kill  points  are  activated  by 
peer  or  untimely  DC  efforts  or  a  misunderstanding  of  the  basic  DC  concepts 
necessary  to  complete  scenario.  These  five  kill  points  are: 

•  The  fire  consumes  a  majority  of  the  ship, 

•  The  Aft  YLS  Magazine  exploded, 

■  Chill  Water  distribution  is  completely  lost, 

•  Vital  cabling  trunk  is  lost,  and 

•  The  YLS  Magazine  is  flooded  without  CO's  permission. 

When  a  program  is  terminated  due  to  kill  point 
activation,  a  message  is  displayed  on  the  Command  and  Control  Console 
specifying  which  kill  point  was  responsible  and  the  actions  taken  or 

neglected  that  caused  its  activation. 

(2)  Pitfalls .  The  IDCTT  Trainer  responds  to  improper 

decisions  with  negative  responses.  Four  conditions  cause  a  message  to 
appear  that  indicates  the  DCA  committed  an  error  or  made  a  poor  decision. 
The  four  pitfalls  are: 

•  The  Bridge  prompts  DCA  for  a  Zebra  report, 

•  The  Ch^ef  Engineer  informs  the  DCA  that  he  must  obtain  permission 
prior  to  starting  a  fire  pump, 

•  The  Commanding  Officer  orders  a  status  report,  and 

•  The  repair  lockers  query  an  incorrect  or  unreasonable  order. 

5 .  Summary 

The  IDCTT  Trainer  p:  ivides  students  with  a  fast  paced  electronic 
training  medium  that  stresses  the  damage-1  control  fundamentals  necessary  to 
complete  a  TSS  based  battle  problem.  The  scenario  provides  students  the 
opportunity  to  exercise  aamage  control  concepts  taught  in  a  simulated 
shipboard  environment.  Stressful  conditions  that  students  face  while 
performing  their  job  as  DCA  aboard  a  ship  are  realistically  replicated. 
In  the  past,  this  training  was  provided  by  the  school's  conventional  DC 


'll 


General  Trainer.  The  f undamentals  of  the  present,  DC  Central  Trainer 
method,  are  outlined  below. 

C.  PRESENT  DAMAGE  CONTROL  TRAINER  TECHNIQUES 

Team  trainers1’  and  simulators  provide  DCAs  with  experience  in 
shipboard  damage  control  before  they  report  aboard  ship.  The  Damage 
Control  Training  Department  currently  uses  trainers  and  simulators  to 
prepare  prospective  DCAs  for  scenarios  they  may  encounter  in  the  fleet. 
The  trainer  used  to  teach  these  scenarios  is  called  the  DC  Central 
Trainer.  Seven  different  lesson  topics  are  taught  to  each  DCA  class  and 
reinforced  through  practical  problems  using  the  DC  Central  Trainer.  These 
topics  include: 

•  Basic  DC  Central  Concepts  Simulator, 

•  Stability  DC  Central  Simulator, 

•  Major  Underwater  Hull  Damage  Simulator, 

•  Main  Space  Fire  Simulator, 

•  Chemical  and  Biological  Warfare  Defense  Simulator,  and 

•  Radiological  Defense  Simulator. 

Given  that  the  DC  Central  Trainer  is  currently  in  use  by  the  School, 
it  provides  an  established  baseline  against  which  to  compare  the  IDCTT 
Trainer's  performance.  To  understand  the  DC  Central  Trainer  method,  a 
description  of  its  objectives  and  an  overview  of  its  implementation  at  the 

Damage  Control  School  are  summarized  below. 

1.  DC  Central  Trainer  Objective 

For  each  training  scenario,  students  spend  one  to  two  hours  in 
the  DC  Central  Trainer  combating  various  subsets  of  problems.  Prior  to 
commencing  the  first  simulator  training  period,  a  90  minute  classroom 
instruction  period  familiarizes  students  with  the  equipment  and  concepts 
of  simulator  team  training. 


■'  Team,  trainers  are  often  used  in  the  Navy  to  train  an  individual  for  a 
cific  whatchstation  in  a  group  environment  that  simulates  shipboard  routine 
special  evolutions. 
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Each  scenario  has  the  same  enabling  and  terminal  objectives.  The 


enabling  objective  states: 

In  the  DC  Central  Trainer,  PERFORM  the  duties  of  one  of  the  following 
personnel  during  a  major  Damage  Control  scenario, 
a  .  DCA 

b.  Plotter 

c.  Sound  Powered  Phone  Talker  (Naval  Education  and  Training 
Command,  Surface  Warfare  Officer  Damage  Control  Assistant  (A-4G-0020 ) 
Course  Curriculum  Outline,  p.4-20,  April  1991). 

The  terminal  objective  states: 

In  the  Damage  Control  Trainer,  DIRECT  the  DC  organization  during 
C'  .eral  Quarters  and  other  emergencies  by  interpreting  and  solving 
Damage  Control  Problems.  ((Naval  Education  and  Training  Command, 
Surface  Warfare  Officer  Damage  Control  Assistant  (A-4G-0020 )  Course 
Curriculum  Outline,  p.4-20,  April  1991). 

The  DC  Trainer  was  originally  developed  as  a  non-computer  based  method  for 

meeting  the  enabling  and  terminal  objectives  stated  in  the  Course 

Curriculum  Outline. 

2.  DC  Central  Trainer  Implementation 

Through  years  of  use  and  modification,  the  manning  and  operation 
of  the  DC  Central  Trainer  has  become  standardized  and  only  slight 
modifications  to  these  aspects  of  the  trainer  were  necessary  to  implement 
the  TSS  based  scenario.16  Unfortunately,  since  the  DC  Central  Trainer's 
scenario  inventory  did  not  contain  a  multiple  weapon  hit  TSS  based 
scenario,  the  development  of  a  scenario  that  could  be  used  for  direct 
comparison  with  the  IDCTT  Trainer  was  dictated.  A  TSS  based  scenario  was 
developed  by  the  author  and  approved  by  the  Damage  Control  School  for  use 
in  the  curriculum.  The  scenario  provided  a  direct  way  to  compare  the  two 
trainers.  Table  1  shows  the  similarities  and  differences  between  the 
IDCTT  and  DC  Central  Trainers.  The  key  issues  of  manning,  operation,  and 

the  scenario  specifics  are  described  below. 

a .  Manning 

The  DC  Central  trainer  was  designed  as  a  group  trainer  to 
provide  students  the  opportunity  to  perform  each  watchstander ' s  duties  in 


•s  The  DCCO  position  was  given  added  responsibility  due  to  the  unique  damage 
control  organization  aboard  an  DDG-51  Class  Destroyer. 
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DC  Central.  A  typical  DCA  class  is  divided  into  groups  of  five  to  seven 
students.  Every  group  performs  each  of  the  seven  training  scenarios  once. 
Within  the  group,  members  rotate  through  the  different  watch  stations.  The 
DC  Central  Trainer  accommodates  a  DCA,  Plotter,  DC  Console  Operator,  and 
two  to  four  Sound  Powered  Phone  Talkers19  depending  on  the  size  of  the 
group.  This  arrangement  enables  each  group  member  to  play  the  role  of  DCA 


TABLE  1:  COMPARISON  OF  SYSTEM  TRAINING  FACTORS  FOR  THE  IDCTT  AND  DC 
CENTRAL  TRAINERS 


Training  Factors 

IDCTT 

DC  Central  Trainer 

Number  of  operators 

One  or  two 

Five  to  seven 

Kill  Points 

Same  as  DC  Central 

Same  as  IDCTT 

Pitfalls 

Same  as  DC  Central 

Same  as  IDCTT 

Scenario  Duration 

Approximately  25 
minutes 

Approximately  90 
minutes 

Instructor 

Directly  monitors 

Physically  removed 

monitoring 

student  actions 

from  student,  monitors 
student  verbal  inputs 

Instructor ' s 

Limited  to  explaining 

Able  to  change 

ability  to  assist 

what  the  IDCTT  is 

scenario  to  fit 

student 

expecting 

students  needs 

Student  inputs 

Drop  down  menus 
provide  students  with 
option  selection 

Students  can  order  any 
actions,  feasible  or 
not 

Scenario  complexity 

Same  as  DC  Central 

Same  as  IDCTT 

White  noise  used  to 
distract  students 

Significant  amounts 

None 

Speed  student  must 
process  information 

Fast 

Slow 

for  at  least  one  scenario.  Although  students  rotate  through  the  various 
watchstations  during  the  six  week  course,  student  groups  only  had  one 
exposure  to  the  DC  Central  Trainer's  TSS  scenario. 


•'  Sound  Powered  Phone  Talkers  can  consolidate  or  separate  phone  lines  to 
support  two,  three  or  four  Phone  Talkers. 
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b.  Operation 


Before  starting  a  DC  Central  Training  session,  the  instructor 
assigns  group  members  to  specific  watch  stations.  The  DCA  then  takes 
charge  of  his  watch  team,  directs  equipment  and  phone  checks,  sets  up  the 
DC  Plates  and  prepares  for  the  scenario  to  begin.  The  DCA  positions 
himself  as  if  he  were  in  DC  Central  with  easy  access  to  the  alarm  panels, 
DC  Plates,  and  Phone  Talkers.  The  Damage  Control  Console  Operator  (DCCO) 
sits  at  the  Damage  Control  Console  while  Phone  Talkers  man  various  sound 
powered  phone  lines. 

The  instructor  provides  all  inputs  to  the  watch  team  using 
a  scripted  scenario.  These  inputs  are  physically  displayed  on  alarm 
panels  which  indicate  fire,  flooding,  changes  in  firemain  pressure,  halon, 
and  carbon  dioxide  release,  opening  of  magazine  sprinklers  and  the  onset 
of  high  temperature  alarms.  Furthermore,  the  instructor  simultaneously 
plays  the  role  of  all  watchstations  which  are  based  outside  of  Damage 
Control  Central.  These  roles  include  the  Captain,  Bridge,  Repairs  II,  III 
and  V,  Combat  Systems  Maintenance  Central  (CSMC) ,  and  any  other  stations 
necessary  to  support  the  scenario.  In  the  present  format,  instructors 
must  monitor  two  or  more  sound  powered  phone  circuits  and  the  21  MC20 
while  indicating  damage  through  a  remotely  operated  alarm  control  panel. 

The  DCA  receives  damage  information  through  the  Sound  Powered 
Phone  Talkers,  alarm  panels,  and  the  21MC.  As  discussed  above,  all 
information  the  DCA  receives  is  generated  by  the  instructor.  The  Plotter 
uses  laminated  DC  Plates  to  assist  the  DCA  by  plotting  all  the  information 
received  and  actions  taken  to  combat  the  damage.  Unlike  the  IDCTT 
Trainer,  the  DC  Central  Trainer  does  not  have  any  computer  display 
monitors  or  DC  Chits.  The  system  relies  on  "hard  wired"  alarm  panels  and 


*'  The  21  MC  is  a  hard  wired,  10  station,  speaker  phone  system.  The  net 
includes  Combat  Information  Center,  Damage  Control  Central,  Radio  Central,  the 
Bridge  and  other  key  locations  throughout  the  ship. 


continuous  dialogue  between  the  DCA  and  the  instructor  to  maintain  a 

continuous  flow  of  information. 

c.  DC  Central  Trainer  Scenario 

Although  current  trainer  manning  levels  and  operational 
techniques  did  not  have  to  be  significantly  modified  for  this  study,  it 
was  necessary  to  develop  a  new  DC  Central  Trainer  scenario.  The  new  DC 
Central  Trainer  scenario  encompassed  the  same  learning  objectives , 
management  options,  kill  points,  pitfalls,  and  emphasis  on  TSS  concepts  as 
the  IDCTT  scer.azio.  Since  students  would  be  exposed  to  both  methods, 
scenario  content  had  to  be  similar  in  difficulty  and  substance  without 
requiring  exactly  the  same  actions  needed  to  successfully  complete  the 
problem.  In  other  words,  if  the  DC  Central  and  IDCTT  Trainer  scenarios 
were  exactly  the  same,  students  would  gain  insight  into  the  scenario  and 
apply  to  whichever  method  they  were  exposed  to  last.  This  insight,  then 
would  confound  the  students'  opinions  and  performance  for  the  two  methods. 

The  scenario  written  for  the  DC  Central  Trainer  employed  the 
same  weapon  types,  one  floating  mine  and  one  Exocet  missile,  as  the  IDCTT 
scenario.  As  in  the  IDCTT  scenario,  the  ship  goes  to  GQ  and  sets  material 
condition  Zebra.  Unlike  the  IDCTT  scenario,  however,  initial  damage  is 
caused  by  an  Exocet  missile  hit  forward  of  the  after  VLS  Magazine.  After 
the  DCA  has  been  given  time  to  react  to  the  initial  damage,  further  damage 
is  inflicted  to  the  ship  by  a  mine  explosion  on  its  port  side.  For  the 
convenience  of  the  reader.  Table  2  compares  the  specific  damage  inflicted 
in  the  IDCTT  and  DC  Central  Trainer  scenarios. 

Similar  to  the  IDCTT  scenario,  damage  spreads  or  is  contained 
depending  on  the  actions  taken  by  the  DCA.  If  the  DCA  reacts  poorly  to 
the  situation,  the  simulation  will  end  with  one  of  the  same  kill  points 
used  in  the  IDCTT  Trainer,  at  the  discretion  of  the  instructor. 
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TABLE  2  :  COMPARISON  OF  WHEN  MAJOR  EVENTS  OCCUR  IN  THE  IDCTT  AND  DC  CENTRAL 
TRAINER'S  SCENARIOS 


Scenario 

Description 

IDCTT  Trainer 

DC  Central  Trainer 

First 

Weapon  Hit 

*  Fire  in  aft  Generator 

Room 

*  Loss  of  one  fire  pump 
and  rupture  to  stbd 
firemam  loop 

'  Loss  of  aft  radar  array 

*  flooding  aft 

*  Personnel  casualties 

*  Fires  forward  of  the  aft 
VLS  Magazine 

*  Rupture  to  port  firemain 
loop 

*  Loss  of  aft  radar  array 

*  Personnel  casualties 

Second 

Weapon  Hit 

’  Fires  on  0-1  level 
'above  and  forward  of 

CIC) 

*  Chill  water  rupture 
'  Personnel  casualties 

*  Fires  on  second  deck 
(below  CIC) 

*  Flooding  in  forward 
compartments 

*  Chill  water  rupture 

*  Personnel  casualties 

3 .  Summary 

Performance  in  the  DC  Central  Trainer  provides  a  standard  for 
comparing  and  determining  that  of  the  IDCTT  Trainer.  Both  trainers 
possess  the  same  fundamental  goal  of  training  DCAs  using  simulated 
multiple  weapon  hit,  TSS  based  training  scenarios.  The  two  systems  differ 
only  in  the  medium  used  to  conduct  this  training. 
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II.  METHODOLOGY 


A .  OVERVIEW 

This  chapter  describes  the  methodology  used  in  the  ICW  validation. 
The  chapter  is  partitioned  into  three  broad  areas.  The  information 
required  to  conduct  the  evaluation  is  outlined  first.  Then  secondly,  the 
method  by  which  data  were  collected  at  the  test  site  is  given.  And  third, 
the  various  statistical  techniques  used  to  reduce  and  analyze  the  data  are 
discussed . 


B.  INFORMATION  REQUIREMENTS 

This  section  discusses  the  salient  information  needed  to  validate  the 
effectiveness  of  the  ICW.  The  validation  process  itself  has  been 
described  elsewhere,  but  for  the  purpose  of  clarification,  the  process  is 
defined  as  follows. 

Validating  the  usefulness  of  instructional  materials  involves 
assessing  the  impacts  on  the  organization  in  relation  to  several 
factors.  These  factors  can  be  classified  into  several  broad 
categories,  including: 

•  Student  achievement; 

•  The  utility,  ease  of  use,  and  creativity  of  the  materials  themselves; 

•  and  Integration  into  the  organization's  instructional  system  (United 
States  Naval  Health  Sciences  Education  and  Training  Command, 
Interactive  Multimedia  Courseware  Validation  Report ,  p.2,  3  December 
1992. )  . 

The  present  validation  effort  examined  the  two  central  areas  reflected  in 
the  definition  cited  above.  These  two  areas  were: 

•  The  IDCTT  Trainer's  performance  in  and  of  itself,  and 

•  The  IDCTT  Trainer  compared  to  the  DC  Central  Trainer. 

Focusing  on  these  areas  produced  the  necessary  information  to  support  a 
comprehensive  ICW  validation.  These  topics  and  the  information  required 

to  support  their  analyses  are  discussed  in  the  following  sections. 

1 .  IDCTT  Trainer  Performance  Evaluation 

The  IDCTT  Trainer  performance  evaluation  was  designed  to  solicit 
student  and  instructor  inputs,  which  taken  together,  would  provide  a 
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subjective  basis  to  evaluate  the  extent  to  which  the  IDCTT  system 
supported  DCA  training.  The  focus  of  this  portion  of  the  study  was  to 
identify  system  strengths,  weaknesses,  and  areas  which  could  be  improved 
to  enhance  the  system  by  modifying  the  prototype .  Also,  measures  of 
student  and  instructor  reactions  to  this  new  training  medium  were  taken  to 
gain  insight  into  the  usefulness  of  the  IDCTT  Trainer  as  a  fleet  training 
aid.  This  information  was  obtained  by  four  relatively  straightforward 
data  collection  techniques.  They  were: 

•  Short  essay  and  narrative  descriptions, 

•  Subjective  rating  data, 

•  Check-off  lists  of  problem  features,  and 

•  Instructor  surveys  and  interviews. 

These  methods  are  each  described  below. 

a.  Short  Essay  and  Narrative  Descriptions 

Short  essay  and  narrative  descriptions  were  solicited  from 
students  after  they  completed  of  the  IDCTT  Trainer  training  period. 
Respondents  evaluated  five  dimensions  of  IDCTT  Trainer  performance  in 
essay  format.  These  five  dimensions  were: 

•  Problem  features  of  the  IDCTT  Trainer,21 

•  The  performance  of  the  touchscreen  monitor, 

•  Problems  encountered  while  using  the  IDCTT  Trainer, 

•  Favorable  aspects  of  the  IDCTT  Trainer,  and 

•  Aspects  of  the  IDCTT  Trainer  that  were  least  liked. 

b.  Subjective  Rating  Data 

Subjective  rating  data  were  obtained  by  administering  three 
surveys  to  students  after  they  completed  IDCTT  training.  Respondents  were 
asked  to  indicate  their  opinion  on  various  aspects  of  the  trainer's 
performance  by  circling  a  number  on  an  eleven  point  rating  scale.  The 
scale  was  anchored  on  both  ends  by  appropriate  language  specific  to  the 

"  This  dimension  provided  amplification  to  the  Check-Off 
problem  features  discussed  later  in  the  text. 
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actual  question  content  and  the  central  value  indicated  a  neutral 
response.  The  three  surveys  used  to  evaluate  this  portion  of  the 
validation  were  the: 

•  Student  IDCTT  Survey, 

•  User  Interface  Dimensions  Questionnaire,  and 

•  Source  of  Workload  Evaluation . 

The  Student  IDCTT  Survey  was  designed  to  obtain  information 
on  unique  aspects  of  the  IDCTT  Trainer.  The  User  Interface  Dimensions 
Questionnaire  was  designed  to  provide  a  standard  approach  in  evaluating 
eight  key  ICW  user  interface  dimensions  (Harmon  and  Reeves,  1993). 
Finally,  the  Source  of  Workload  Evaluation,  a  technique  developed  by  NASA, 
was  used  to  assess  the  relative  importance  of  six  factors  with  regard  to 
how  much  workload  students  experienced  while  operating  the  ICW  (Hart, 

1988) .  The  three  surveys  appear  in  their  entirety  in  Appendix  A. 
c.  Check-Off  Problem  features 

After  completing  the  IDCTT  training  session,  students  were 
presented  with  a  list  of  IDCTT  Trainer  hardware  and  software  features  and 
asked  to  indicate  which  feature,  if  any,  caused  them  difficulty.  These 
features  were  generated  from  potential  problem  areas  identified  by  the 
system  developers  at  CIM.22  The  following  seven  features  were  evaluated: 

•  The  operation  of  the  touchscreen  monitor, 

•  The  clarity  of  audio  reports, 

•  The  ability  to  find  DC  Plate  information, 

•  The  acceptability  of  the  speed  or  volume  of  information  presented, 

•  The  presentation  of  the  Damage  Control  Alarm  Panel  display, 

•  The  presentation  and  operation  of  the  Firemain  Alarm  Panel  display, 
and 

•  Other  features  not  included  in  the  survey  list. 


The  project  development  team  and  the  author  compiled  these  topics  based 
or.  their  own  experience  using  the  IDCTT  Trainer. 
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Students  were  invited  to  write  about  any  difficulties  they  experienced  on 

these  items  in  space  provided  on  the  survey  instrument. 

d.  Instructor  Evaluation 

After  completing  the  training  session,  each  instructor  was 
interviewed  and  asked  to  complete  a  survey  which  evaluated  the  efficacy  of 
the  system  as  a  training  aid.  The  interview  was  open  ended,  allowing 
instructors  to  describe  their  impressions  of  the  system  and  its  usefulness 
as  a  shore  based  training  aid.  The  Instructor  IDCTT  Survey  produced 
ranking  data  and  short  narratives.  The  survey  evaluated  the  instructors' 
ability  to: 

•  Use  the  IDCTT  system  as  a  training  aid,  and 

•  To  critique  students'  performance. 

2 .  IDCTT  Versus  DC  Central  Trainer  Performance  Comparison 

A  direct  comparison  of  the  IDCTT  Trainer  with  respect  to  the  DC 
Central  Trainer  was  done  to  determine  the  relative  effectiveness  of  both 
trainers.  The  evaluation  sought  to  draw  comparisons  by  obtaining  student 
performance  scores  for  each  trainer  type  on  a  variety  of  comparative 
dimensions,  and  on  measures  of  each  systems'  ability  to  produce 
standardized  training  scenarios.  This  information  was  obtained  through: 

•  Graded  student  sessions,  and 

•  Subjective  rating  data. 

These  measures  are  described  below. 

a.  Graded  Student  Sessions 

Each  method  provided  the  student  with  the  same  information 
needed  to  complete  the  damage  control  scenario.  Since  the  scenarios  were 
designed  to  be  equally  difficult,  student  performance  primarily  depended 
on  the  medium  in  which  they  operated.  A  standardized  scoring  system  was 
developed  for  assigning  grades  to  student  performance  in  each  trainer, 
'■'hen  compared,  these  scores  provided  a  baseline  measure  of  the  IDCTT' s 
effectiveness  against  the  present  standard;  that  is,  performance  in  the  DC 
Central  Trainer. 
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Prior  to  the  present  study,  DC  Central  Trainer  sessions  were 
ungraded.  The  Damage  Control  School's  policy  was  to  simply  expose  DCAs  to 
damage  control  situations  and  not  formally  grade  them.  To  facilitate 
quantitative  basis  on  which  to  compare  students'  performance  between  the 
two  methods,  specified  grading  criteria  were  developed  with  assistance 
from  instructor  experts  at  the  Damage  Control  School.  The  criteria  were 
modeled  after  Fleet  Exercise  Publication  Number  4  (FXP-4)2j  grading 
standards  which  are  used  to  grade  damage  control  organizations  in  the 
fleet.  The  grading  criteria  developed  for  the  present  evaluation 
emphasized  basic  damage  control  principles,  TSS  concepts,  asset 
management,  and  common  sense.  The  same  grading  criteria,  a  complete  copy 
of  which  is  provided  in  Appendix  A,  were  used  for  both  the  IDCTT  and  DC 

Central  Trainers. 

b.  Subjective  Rating  Data 

Subject d"e  rating  data  that  compared  aspects  of  the  two 
systems  were  obtained  using  two  separate  surveys.  The  two  surveys  were 
the : 

•  IDCTT  Versus  DC  Central  Trainer  Comparison  Survey,  and  the 

•  Scenario  Topics  Ranking  Survey. 

The  Trainer  Comparison  Survey  requested  students  compare  the  two  systems 
across  eleven  different  dimensions.  The  Scenario  Topics  Ranking  Survey 
measured  students'  perceptions  regarding  the  consistency  with  which  each 
trainer  met  the  specified  damage  control  learning  objectives.  These  two 
surveys  are  described  below  and  are  included  in  Appendix  A  for  the 
interested  reader . 

(1)  IDCTT  Versus  DC  Central  Trainer  Comparison  Survey. 
After  students  were  exposed  to  both  training  environments,  they  completed 
a  survey  which  was  designed  to  evaluate  the  IDCTT  Trainer  with  respect  to 
the  DC  Central  Trainer.  Eleven  system  characteristics  were  presented  on 


lj  FXP-4  provides  Afloat  Training  Organizations  with  a  grading  criteria  for 
various  damage  control  problems  while  providing  ships  with  a  training  guideline. 
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a  bipolar  preference  scale  anchored  on  one  end  by  the  IDCTT  and  on  the 
other  by  the  DC  Central  Trainer.  A  “six*  on  the  scale  indicated  “no 
preference"  between  the  two  methods.  Students  circled  the  number  which 
corresponded  to  the  degree  they  felt  one  method  outperformed  the  other. 
This  item  series  attempted  to  determine  which  of  the  two  systems  better 
met  the  following  eleven  criteria: 

»  Simulated  the  shipboard  environment  more  realistically, 

•  Enabled  instructors  to  provide  complete  post  scenario  debriefs, 

•  Produced  the  greatest  level  of  stress, 

•  Allowed  instructors  to  monitor  student's  performance, 

•  Prepared  the  student  for  actual  shipboard  emergencies, 

•  Updated  student  inputs  more  easily, 

•  Provided  scenario  information  closely  resembling  shipboard  methods, 

•  Provided  more  effective  teaching  environment  to  exercise  damage 
control  skills, 

•  Promoted  greater  learning  in  the  time  allotted, 

•  Preferred  training  method,  and 

•  Stimulated  the  student  to  perform. 

Responses  from  this  bipolar  ranking  series  was  used  to  determine  how  the 

two  methods  compared  across  the  range  of  capabilities  listed  above. 

(2)  Scenario  Topics  Ranking  Survey.  The  Scenario  Topics 

Ranking  Survey  was  administered  after  students  completed  each  training 
method  to  measure  the  extent  to  which  each  trainer  delivered  a 
standardized  scenario.  The  survey  listed  a  series  of  fundamental  damage 
control  actions  needed  to  successfully  complete  a  damage  control  problem. 
Students  rated  the  extent  to  which  each  fundamental  topic  played  a  role  in 
the  battle  problem  delivered  by  the  IDCTT  and  DC  Central  Trainers. 

The  Scenario  Topics  Ranking  Survey  yielded  two  measures. 
First,  an  ordinal  ranking  of  the  13  damage  control  fundamentals  each 
scenario  emphasized  was  made  based  on  the  median  scale  value  from  the 
subjective  rating  responses.  This  ranking  was  used  to  determine  if  the 
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two  trainers  emphasized  the  same  damage  control  fundamentals,  and  using 
the  median  responses,  to  what  degree.  Second,  the  interquartile  range 
i  IQR:  from  each  trainer's  rating  data  was  compared  to  determine  how  much 
the  student  ratings  varied  across  the  two  trainers.  These  data 
r.ighl  ignted  the  extent  to  which  each  system  consistently  emphasized  the 
same  learning  objectives  from  the  perspective  of  the  student. 

C.  DATA  COLLECTION  METHODS 

The  IDCTT  Trainer  validation  study  was  conducted  at  the  Damage  Control 
Training  Department  of  the  Surface  Warfare  Officer  School  in  Newport, 
Rhode  Island.  This  school  is  the  sole  source  of  academic  training  for 
novice  DCAs .  Before  this  study,  the  school's  only  source  of  simulated  DCA 
battle  problem  training  was  the  DC  Central  Trainer.  The  new  IDCTT  Trainer 
was  transported  to  the  DCA  School  and  installed  in  it's  simulation  wing 
for  the  purpose  of  this  validation  study. 

Data  were  collected  on  three  occasions.  The  first  test  period,  24 
through  29  March  1994,  consisted  of  trials  used  to  test  the  data 
collection  methodology  and  the  IDCTT  Trainer's  performance  characteristics 
in  a  field  environment.  The  second  and  third  test  periods,  conducted  19 
through  21  April  1994,  and  20  through  22  June  1994,  were  used  to  collect 
the  actual  validation  data.  The  three  test  dates  were  scheduled  during 
the  last  week  of  each  DCA  class  to  ensure  students  had  the  knowledge 
necessary  to  complete  complex  damage  control  scenarios,  such  as  those 
which  would  be  presented  in  the  IDCTT. 

As  discussed  in  the  previous  section,  the  data  used  in  this  validation 
was  collected  by  a  series  of  surveys,  interviews,  and  graded  performance 
evaluations.  Data  were  collected  after  students  completed  training  in 
each  type  device.  Comparison  data;  that  is,  data  that  were  ultimately 
used  to  compare  and  contrast  the  technical  features  of  both  training 
devices,  were  collected  upon  completion  of  the  entire  training  evolution. 
Instructor  data  were  collected  after  the  student  training  sessions  were 
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completed.  Table  3  shows  the  type  of  data  solicited  at  various  stages  of 


the  data  collection  period.  The  following  sections  describe  student 
assignments  for  training,  survey  administration,  student  scoring  criteria, 
and  the  pilot  test  period  used  to  finalize  the  validation  methodology. 


TABLE  3 :  DATA  COMPLETION  POINTS  DURING  THE  EVALUATION  STUDY  PERIOD 


Sources  of  Data 

Data  Collected 

IDCTT  Trainer 

1 .  Student  IDCTT  Trainer  Survey 

2.  Scenario  Topics  Survey  (for  IDCTT) 

3.  Source-of -Workload  Evaluation 

4.  User  Interface  Dimensions  Questionnaire 

5 .  Student  Grade  Sheet 

DC  Central  Trainer 

1.  Scenario  Topics  Survey  (for  DC  Central 

Trainer) 

2 .  Student  Grade  Sheet 

3oth  Trainers 

1.  Student  IDCTT  vs  DC  Central  Trainer 

Comparison  Survey 

Student 

Participation 

1 .  Instructor  IDCTT  Survey 

2 .  Instructor  Interviews 

1.  Student  Assignments 

Students  were  in  their  final  week  of  damage  control  school  when 
they  participated  in  the  validation  study.  Three  class  days  were 
scheduled  for  each  of  the  data  collection  periods.  During  each  of  these 
periods,  students  were  randomly  assigned  in  groups  of  two  for  the  IDCTT 
Trainer,  and  groups  of  five  to  seven  for  the  DC  Central  Trainer.  For  each 
of  the  test  periods,  a  class  of  30  students  were  scheduled  to  participate 
ir.  the  study.  However,  due  to  scheduling  conflicts  and  time  limitations, 
only  a  portion  of  each  group  completed  the  entire  evolution.  The 
following  guidelines  were  used  to  schedule  the  students  through  tne  two 
training  methods: 

•  Half  of  the  students  were  assigned  to  the  IDCTT  Trainer  first  while 
the  other  half  used  the  DC  Central  Trainer.  This  procedure  ensured 
each  trainer  had  the  same  number  of  students  exposed  to  its  style 
first. 

•  Each  of  the  two  students  assigned  to  the  IDCTT  Trainer  was  given  one 
practice  session  and  ore  graded  session.  Furthermore,  The  two 
students  in  tne  group  exchanged  duties  as  DCA  and  Plotter  for  a  total 
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of  two  practice  sessions  followed  by  two  graded  sessions  in  one 
training  period. 

•  The  following  four  position  sequences  were  randomly  assigned  to 
students  performing  in  the  IDCTT  Trainer: 

1.  DCA-Flotter-DCA-Plotter, 

2.  Piotter-DCA-Plotter-DCA, 

3.  Plotter -DCA-DCA-Plotter ,  and 

4.  DCA-Plotter-Plotter-DCA. 

•  The  student  DCA  for  the  DC  Central  Trainer  was  randomly  selected  from 
the  group  of  five  to  seven  students  for  each  training  group. 

•  Student  groups  were  scheduled  for  90  minute  training  periods  for  each 
training  method. 

2 .  Survey  Administration 

Before  a  survey  was  distributed,  its  instructions  were  read 
aloud,  and  students  were  encouraged  to: 

•  Seek  clarification  on  parts  of  the  survey  they  did  not  understand, 
and 

•  Comment  on  items  they  felt  were  important,  but  were  not  included  in 
the  survey. 

The  order  in  which  students  received  the  six  different  surveys  depended  on 
which  training  method  they  were  exposed  to  first.  Surveys  were 
distributed  according  to  the  guidelines  shown  in  Table  3.  Each  survey  was 
given  to  the  students  immediately  after  a  training  session  (IDCTT  Trainer, 
DC  Central  Trainer,  or  Both) .  Both  the  survey  coordinator  and  instructors 
were  available  to  clarify  any  misunderstandings  the  students  had  on  the 
surveys .  Instructor  surveys  were  distributed  after  students  completed 

their  training  sessions. 

3  .  Student  Scoring  Criteria 

Student  performance  was  monitored  and  graded  in  both  the  IDCTT 
and  DC  Central  Trainers.  Grades  were  assigned  based  on  students' 
demonstrated  level  of  proficiency  during  the  90  minute  training  period. 
Grade  assignment^  were  standardized  using  the  Student  Grade  Sheet  criteria 
described  earlier.  This  grading  procedure  provided  student  data  for  one 
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graded  IDCTT  Trainer  session  and  one  graded  DC  Central  Trainer  session.24 
It  should  be  noted,  however,  that  although  the  same  grading  criteria  were 


used  for  both  trainers,  the  ability  of  the  instructor  to  monitor  the 
students'  performance  varied  between  trainers.  The  following  sections 

describe  how  instructors  graded  the  students  for  each  trainer  type. 

a.  IDCTT  Trainer  Grading  Protocol 

Each  of  the  two  students  assigned  to  the  IDCTT  Trainer 
completed  one  practice  session  as  the  DCA  and  one  as  the  Plotter. 
Students  were  encouraged  to  ask  questions  about  any  aspect  of  the  trainer 
they  did  not  fully  understand.  After  their  questions  were  answered,  and 
they  reported  feeling  comfortable  using  the  new  trainer,  each  student 
completed  one  IDCTT  Trainer  scenario  as  the  DCA  and  was  assigned  a 
quantitative  grade  that  reflected  their  level  of  proficiency.  Instructors 
observed  the  students'  progression  through  the  scenario  from  inside  the 
IDCTT  Trainer  module  room.  This  enabled  instructors  to  visually  monitor 
student  input  into  the  IDCTT' s  Command  Console,  and  to  observe  the 
students'  reactions  to  the  various  stressful  situations.  From  this 
vantage  point,  instructors  assigned  grades  on  a  scale  from  one  to  100, 
based  on  the  criteria  specified  on  the  Student  Grade  Sheet.  Points  were 
deducted  depending  on  the  percentage  of  each  grading  topic  not 
completed . 25 

b.  DC  Central  Trainer  Grading  Protocol 

All  students  who  participated  in  the  evaluation  received 
training  throughout  the  DCA  School's  curriculum  in  DC  Central  Trainer 
operations.  Thus,  students  were  assumed  to  be  proficient  in  this 
particular  trainer  before  the  day  of  the  actual  test  session.  Student 


Before  the  validation  test  date,  students  had  completed  seven  different 
DC  Central  Trainer  scenarios  which  were  required  for  graduation  from  the  DCA 
School.  Thus,  students  were  already  skilled  in  the  use  of  this  method. 

It  should  be  noted  that  although  the  IDCTT  Trainer  has  a  playback  feature 
fcr  post  scenario  debriefs,  this  feature  was  not  used  due  to  time  constraints  in 

c  T  si  J  *5  ri  c  s  c  h  ^  ci  si  —  i  n  g  . 
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groups  were  scheduled  for  90  minutes  of  DC  Central  Trainer  simulation 
training  using  the  Total  Ship  Survivability  based  battle  problem. 
Students  in  this  group  were  randomly  assigned  to  the  DCA  and  other 
positions  by  the  instructor.  The  simulation  was  run  once,  and  took  the 
entire  90  minute  training  period  to  complete.  As  with  the  IDCTT  Trainer, 
instructors  graded  student's  performance  based  on  the  standardized  grading 
criteria . 

One  aspect  of  the  DC  Central  Trainer  that  deserves  comment; 
that  is,  in  this  trainer,  the  instructor  is  physically  isolated  from  the 
students  during  the  exercise.  Since  only  one  instructor  was  assigned  to 
each  group,  there  was  no  direct  means  to  evaluate  the  students' 
performance  except  from  the  inputs  the  instructor  received  from  the  other 
side  of  the  wall.  Unlike  the  IDCTT  Trainer  which  recorded  the  entire 
training  session  and  was  designed  to  accommodate  an  instructor  in  the 
room,  an  error  in  the  DC  Central  Trainer  was  lost  if  it  was  undetected  in 
the  voice  communications .  Generally,  this  arrangement  would  bias  test 
results  toward  better  performance  in  the  DC  Central  Trainer  because  some 

mistakes  would  go  unobserved  causing  higher  performance  scores. 

4 .  Data  Collection  Pilot  Test  Period 

Initially,  the  data  were  scheduled  to  be  collected  during  24 
through  29  March  1994.  This  test  period  was  the  first  time  IDCTT  was  used 
outside  the  development  facility.  This  pilot  test  period  was  used  to  test 
the  system  hardware  and  software  performance  in  its  intended  operating 
environment,  and  to  verify  and  standardize  the  data  collection 

methodology . 

a.  Schedule 

As  discussed  above,  the  validation  series  lasted  three  days, 
during  -which  30  students  were  scheduled  to  participate  in  the  study.  All 
20  students  completed  the  DC  Central  Trainer  portion  of  the  test  sequence. 
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but  due  to  frequent  “crashes”  of  the  IDCTT  Trainer,  only  12  students 
completed  the  IDCTT  evaluation  phase.24 

The  original  test  plan  required  each  student  to  complete  two 
practice  sessions  in  the  IDCTT  Trainer  followed  by  a  third  session  which 
was  graded.  This  procedure  took  about  90  minutes  of  single  user  IDCTT 
Trainer  time  per  student;  approximately  the  same  amount  of  time  as  one  DC 
Central  Trainer  scenario.  Due  to  the  limited  time  students  were  available 
for  testing,  this  requirement  was  revised  to  one  practice  session  followed 
by  a  graded  session.  Further,  original  test  design  required  the  student 
to  operate  the  IDCTT  Trainer  in  the  single  user  mode,  acting  as  their  own 
plotter  by  manipulating  the  ISMS.  This  procedure  was  changed  because 
students  spent  excessive  time  trying  to  find  information  using  the  ISMS, 
which  distracted  their  attention  from  the  main  battle  problem.  To  offset 
this  attentional  diversion,  a  second  student  was  added  to  operate  the  ISMS 
and  perform  plotting  duties. 

The  30  student  participants  completed  their  scheduled  IDCTT 
practice  runs,  but  unexpectedly,  this  took  the  entire  allocated  testing 
period.  The  school  was  unable  to  accommodate  the  allocation  of  further 
testing  time  for  the  purpose  of  this  pilot  study  due  to  its  existing  class 
schedules.  The  school  permitted  continued  testing  for  the  graded 
sessions,  after  hours,  on  a  voluntary  basis.  Twelve  of  the  30  students 

volunteered  to  participate  in  this  follow-on  testing. 

b.  ISMS  Removed 

For  the  graded  sessions,  ISMS  was  dropped  from  the  IDCTT 
Trainer  system  and  replaced  with  DC  Plates.  ISMS  was  deleted  for  three 
reasons : 

•  Students  had  not  been  previously  exposed  to  the  ISMS  system,  thus 
requiring  them  to  learn  the  two  systems  simultaneously. 


‘ '  During  the  pilot  test  period,  a  system  "crash"  occurred  approximately  one 
time  in  every  six  training  sessions.  Once  a  "crash"  occurred,  the  system  had  to 
be  soft  booted  to  reinitialize  the  training  program.  This  procedure  lost  the 
previous  scenario,  thus  requiring  the  student  to  start  from  the  beginning. 


•  Students  could  not  locate  specific  valve  numbers  from  the  piping 
diagrams  due  to  the  small  scale  representation. 

•  Software  interface  problems  between  the  486  computer  and  the  ISMS 
system  frequently  caused  the  system  to  lock-up. 

The  ISMS  syscem  therefore  was  deleted  from  the  IDCTT  Trainer  throughout 

the  remainder  of  the  validation  study. 

c.  IDCTT  Trainer  System  Improvements 

Upon  completing  the  pilot  test  period,  the  IDCTT  Trainer  was 
sr.  to  CIM  for  software  updates  based  on  the  information  obtained 

d.  the  pilot  tests.  These  updates  will  be  discussed  later,  but 

suffice  it  to  say  here  that  the  revised  version  eliminated  system  crashes 
and  improved  the  user  interface  and  increased  the  likelihood  that  students 

would  complete  the  scenario. 

d.  Summary 

From  the  information  obtained  during  the  pilot  test  period, 
Che  data  collection  methodology  actually  used  was  only  slightly  modified. 
Further,  the  prototype  IDCTT  Trainer  was  also  modified  based  on 
information  collected  during  the  initial  tests.  After  these  changes  were 
implemented,  the  IDCTT  Trainer  version  was  not  modified  during  the 
remainder  of  the  study. 

D.  DATA  ANALYSIS  METHODS 

Upon  completion  of  each  test  trial,  all  data  were  sorted  and  placed 
into  individual  student  data  packages.  A  student  file  contained  the  four 
data  records : 

•  Short  essay  and  narrative  descriptions, 

•  Subjective  Rating  Data, 

•  Frequency  data,27  and 

•  Performance  grades 


Frequency  data  were  collected  on  the  percentage  of  students  who  felt 
various  features  of  the  IDCTT  Trainer  required  improvements. 
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These  data  were  then  compiled  to  accommodate  the  statistical  analyses 
employed  by  this  study.  These  statistical  methods  are  described  in  the 
following  sections. 


1.  Short  Essay  and  Narrative  Descriptions 

Short  essay  and  narrative  descriptions  were  included  in  the 
validation  study  to  augment  the  rating  data  and  frequency  responses. 
Because  of  the  qualitative  nature  of  the  narratives,  summary  statistics 
could  not  be  computed,  but  similar  student  responses  were  categorized, 
then  summarized  and  listed  for  each  subject.  These  listings  provided  a 
means,  although  subjective,  to  isolate  patterns  in  student  responses  that 

further  explicated  their  quantitative  responses. 

2.  Rating  Data 

The  rating  data  were  analyzed  using  the  Method  of  Equal-Appearing 
Intervals  (MEAIS )  (Thurstone  and  Chave,  1929).  Using  this  method,  the 
scale  data  obtained  from  the  student  questionnaires  were  compiled  and 
placed  in  summary  tables  similar  to  that  shown  in  Table  4. 


TABLE  4:  EXAMPLE  MEAIS  DATA  MATRIX 


Statement 

1 

2 

3 

4 

Sorting  categories 

5  6  7  8  9 

10 

11 

f 

4 

4 

5 

7 

4 

3 

2 

1 

2 

0 

0 

1 .  p 

.  13 

.13 

.  16 

.22 

.  13 

.09 

.06 

.03 

.06 

.00 

.00 

cp 

.  13 

.26 

.41 

.63 

.75 

.84 

.91 

.94 

1.0 

1.0 

1.00 

For  each  statement,  three  rows  were  used  to  display  summary  statistics 
from  the  student  data.  The  first  row  (f),  indicated  the  total  frequency 
that  students  circled  each  number.  The  second  row  (p)  ,  displayed  the 
frequency  as  a  proportion  to  the  total  number  of  students.  The  final  row 
(cp),  cumulatively  summed  the  proportion  of  frequency  that  had  occurred 
through  the  numeric  values. 

If  the  median  of  the  distribution  of  judgments  for  each  statement  is 
taken  as  the  scale  value  of  the  statement,  then  the  scale  values  can 
be  found  from  the  data  arranged  in  the  manner  of  Table  4  by  means  of 
the  following  formula: 
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S=L+ ( . 50  -5>*>  *PW)  i  • 


where  S  =  the  median  or  scale  value  of  the  statement 

L  =  the  lower  limit  of  the  interval  in  which  the 
median  falls 

£pc  =  the  sum  of  the  proportions  below  the 
interval  in  which  the  median  falls 
pw  =  the  proportion  within  the  interval  in  which 
the  median  falls 

i  =  the  width  of  the  interval  and  is  assumed  to  be 
equal  to  1.0  {Edwards,  p87,  1957). 

The  interquartile  range  provided  a  measure  of  the  variability  in 
the  distribution  of  student  responses.  The  interquartile  range  (Q  or  IQR) 
provided  a  numeric  value  representing  the  range  of  numbers  in  which  the 
middle  50  percent  of  the  scale  judgments  fell.  To  find  Q,  the  25th  and 
75th  percentiles  were  calculated  and  subtracted  using  the  following 
formulas : 


C25=L+  (  .25  -Y,Pb)*Pw)i, 


where  C2i  =  the  25th  percentile  of  the  statement 

L  =  the  lower  limit  of  the  interval  in  which  the  25th  percentile 
falls 

Xpb  =  the  sum  of  the  proportions  below  the  interval  in  which 
the  25th  percentile  falls 

ptt  =  the  proportion  within  the  interval  in  which  the  25th 
percentile  falls 

i  =  the  width  of  the  interval  and  is  assumed  to  be  equal  to  1.0 
(Edwards,  p89,  1957); 


C75=L+(. 75  “£  Pb)*Pw)i' 


where  C,5  =  the  75th  percentile  of  the  statement 

L  =  the  lower  limit  of  the  interval  in  which  the  75th  percentile 
falls 

Xpb  =  the  sum  of  the  proportions  below  the  interval  in  which  the 
75th  percentile  falls 

pw  =  the  proportion  within  the  interval  in  which  the  75th 
percentile  falls 

i  =  the  width  of  the  interval  and  is  assumed  to  be  equal  to  1.0 
(Edwards,  p89,  1957); 


^  *-7  5  ^"25  " 
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Once  the  scale  value  (S)  and  the  interquartile  range  (Q)  were  calculated 
for  each  statement,  summary  statistics  indicating  students  reaction  to  the 

various  items  were  compiled  for  further  analysis. 

3 .  Frequency  Data 

Frequency  data  were  analyzed  using  the  proportion  of  students  who 
responded  to  each  check-list  item.  This  summary  statistic  was  obtained  by 
summing  the  number  of  students  who  checked  each  item  and  dividing  by  the 
total  number  of  students  in  the  study.  This  information  provided  a 

percentage  of  students  who  agreed  on  each  checklist  item. 

4.  Performance  Grades 

The  significance  of  the  differences  between  paired  performance 

grades  for  the  IDCTT  and  DC  Central  Trainer  was  evaluated  using  the  Large- 

Sample  Wilcuxon  Signed-Rank  Test  tor  a  Paired  Experiment  (Mendenhall, 

1990).  This  test  determines  if  the  relative  frequency  distribution  for 

the  two  sets  of  grades  are  identical  or  different. 

The  following  summarizes  the  testing  procedure. 

To  carry  out  the  Wilcoxon  test,  the  Differences  for  each  of  the 
paired  scores  is  calculated.  Differences  equal  to  zero  are 
eliminated.  The  rank  of  the  absolute  values  for  each  of  the 
numbers  is  determined,  assigning  a  1  to  the  smallest,  2  to  the 
second  smallest  and  so  on.  The  rank  sum  is  calculated  for  each 
the  positive  and  negative  differences.  The  positive  value  of 
these  two  calculations  is  used  to  calculate  the  z  value  from  the 
normal  curve  and  is  used  as  the  test  statistic  the  test 
statistic.28  This  test  statistic  is  then  compared  against  the 
z  value  for  the  appropriate  significance  '■ evel  desired.  This 
comparison  is  used  to  determine  if  the  null  hypothesis,  that  the 
two  frequency  distributions  are  the  same,  should  be  accepted 
(Mendenhall,  p.680,  1990). 

The  results  of  this  test  provided  a  means  for  evaluating  whether  the 
trainers  were  equally  effective  as  a  student  training  aid. 

E.  SUMMARY 

The  pilot  test  period  indicated  that  the  validation  test  plan  was 
practicable,  with  only  slight  modifications.  As  a  result  of  the  pilot 
test,  the  prototype  trainer  underwent  only  minor  software  updating,  and 

:e  See  Appendix  C  for  the  z  value  equation. 


the  ISMS  system  was  dropped  from  the  IDCTT  Trainer.  Accordingly,  a 
summary  of  the  validation  methodology  is  displayed  in  Figure  1  for  the 
convenience  of  the  reader. 
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III.  RESULTS 


This  chapter  presents  the  results  obtained  from  the  validation 
effort's  experimental  group.  The  validation  experimental  group  consisted 
of  32  students  from  two  separate  classes.  As  previously  stated,  data  were 
collected  on  a  total  of  32  students;  22  students  who  were  tested  between 
19  and  21  April  1994,  and  10  students  who  were  tested  between  20  and  22 
June  1994.  Students  in  this  test  group  represented  a  cross  section  of  the 
average  classes'  fleet  experience  level,  and  damage  control  aptitude 
(Jullian,  1994) . 

The  data  are  presented  in  two  sections.  The  first  section  presents 
the  results  of  the  IDCTT  Trainer  performance  evaluation;  that  is,  the 
evaluation  of  the  IDCTT  Trainer  alone.  The  second  section  provides  the 
results  of  the  IDCTT  versus  DC  Central  Trainer  performance  comparison.  No 
attempt  is  made  to  provide  a  substantial  discussion  of  the  results  in  this 
chapter.  That  will  take  place  in  the  next  chapter. 

A.  IDCTT  TRAINER  PERFORMANCE  EVALUATION 

Again,  data  were  collected  from  32  students  undergoing  instruction  and 
the  eight  Damage  Control  School  instructors  to  evaluate  the  IDCTT  Trainer. 
These  data  were  comprised  of  short  essay  and  narrative  descriptions, 
rating  data,  check-off  problem  features,  and  instructor  evaluations. 

These  results  are  presented  in  the  following  four  sections. 

1.  Short  Essay  and  Narrative  Descriptions 

Short  essay  and  narrative  descriptions  were  solicited  from 
students  using  the  Student  IDCTT  Trainer  Survey.  Each  student  responded 
to  five  essay  questions.  The  first  question  which  asked  students  to 
identify  and  explain  any  IDCTT  Trainer  feature  which  caused  them 
difficulties  while  interacting  in  the  IDCTT  training  environment,  are 
summarized  in  Table  5.  The  final  four  questions,  which  also  dealt  with 
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A  compilation  of 


IDCTT  features,  are  similarly  summarized  in  Table  6. 29 
all  the  students'  responses  is  included  in  Appendix  B 


TABLE  5 :  STUDENT  RESPONSES  TO  A  LIST  OF  IDCTT  TRAINER  FEATURES  IN  WHICH 
THEY  WERE  ASKED  TO  EXPLAIN  WHICH  FEATURES  CAUSED  THEM  DIFFICULTY  WHILE 
OPERATING  THE  SYSTEM 


Feature 

Comnents 

Touchscreen 

Monitor 

1.  Touchscreen  is  not  sensitive  to  the  touch  and 
responds  slowly  to  student  inputs. 

2.  The  screen  displayed  options  which  can  not  be 
activated  or  serve  no  purpose. 

3 .  Option  selections  buttons  are  too  close  together 
causing  accidental  inputs  into  the  IDCTT 

Trainer . 

4.  Students  were  unfamiliar  with  which  option 
selection  was  necessary  to  initiate  specific 
actions . 

Understanding 
Audio  Reports 

1 .  Background  audio  track  contained  clearly 
erroneous  information. 

2.  Unable  to  ask  computer  simulated  watchstanders 
to  repeat  previously  given  information. 

Finding  DC 

Plate 

Information 

1.  Student  unfamiliarity  with  the  design  of  the 

DDG-51  Class  Destroyer  made  locating  information 
from  it's  DC  plates  difficult. 

2.  Lack  of  practice  in  using  DC  Plates  made  it 
difficult  co  find  information  from  the  DC  Plates 
within  the  limited  time  allotted  in  the  quick 
pace  of  the  IDCTT  scenario. 

Speed  or  volume 
of  information 
presented 

1.  The  st'-r  -o  was  too  fast  for  beginners,  the 
scenai ,  ?  a  should  be  adjustable  to  meet  the 

student's  level  of  expertise. 

DC  Alarm  Panel 
Display 

No  Responses 

Fi remain  Panel 
and  firemain 
valve  and  pump 
operations 

1.  The  fireuain  Alarm  Panel  Screen  was  too  small 
making  it  difficult  to  read  valve  numbers  and 
labels . 

Other  features 
not  listed 

1.  Unfamiliarity  with  the  system  made  it  initially 
difficult  to  operate. 

2.  System  is  unresponsive  to  student  inputs  to 
activate  all  relevant  valves  or  investigate  and 
combat  damage  in  some  of  the  damaged 
compartments . 

3.  Unable  to  obtain  specific  information  on 
integrity  of  the  chill  water  system. 

29  Summary  statements  are  based  on  a  compilation  of  written  and  oral 
responses  selected  as  a  representation  of  the  students'  responses.  A  complete 
listing  of  all  student  responses  is  included  in  Appendix  B. 


TABLE  6:  SUMMARY  OF  STUDENTS'  SHORT  ESSAY  AND  NARRATIVE  RESPONSES  TO 


STUDENT  IDCTT  TRAINER  SURVEY  NARRATIVE  QUESTIONS 


Question 

Comments 

How  can  the 
touchscreen 
control  panel 
be  improved? 

1.  Increase  the  size  of  the  number  pad  on  the 
touchscreen  or  only  use  the  computer  keyboard. 

2.  Increase  the  touchscreen  sensitivity. 

3.  Use  a  mouse  to  select  options  or  switch  to  voice 
activation  technology. 

4.  Eliminate  the  touchscreen  options  which  are  not 
activated . 

What  problems 
did  you 

encounter  while 
using  the  IDCTT 
Trainer? 

1.  Input ing  information  in  the  Command  Console 
touchscreen  monitor. 

2.  Unfamiliarity  with  the  DDG-51  Class  Destroyer 
made  it  difficult  to  visualize  what  the 
information  received  meant. 

3.  Not  fully  understanding  what  each  button  option 
would  do  stemming  from  inexperience  in  using  the 
system. 

4.  Inexperience  in  making  reports  to  the  Commanding 
Officer,  while  the  system  provided  no  clear 
means  to  report  to  him. 

5.  Little  or  no  knowledge  of  the  chill  water 
system. 

6.  System  occasionally  locked-up  for  no  apparent 
reason . 

What  aspects  of 
the  IDCTT 

Trainer  did  you 
like  the  most? 

1.  The  trainer  was  extremely  realistic. 

2.  The  amount  of  stress  induced  by  the  system. 

3.  Ability  for  one  person  to  train  without  the  need 
for  an  entire  watch  team. 

4.  Clarity  of  information  the  IDCTT  Trainer 
presented. 

5.  Fast  pace  and  audio-visual  presentation  of  the 
information . 

6.  Printed  damage  control  chits. 

7 .  Enables  the  user  to  observe  the  consequences  of 
their  actions  while  providing  the  option  to 
repeat  the  scenario  until  it  is  done  correctly. 

What  aspects  of 
the  IDCTT 

Trainer  did  you 
like  the  least 

1.  Only  one  scenario  for  only  one  ship  class. 

2.  Difficulties  experienced  while  inputing 
information  into  the  touchscreen  monitor. 

3.  The  emphasis  on  the  chill  water  system  when  the 
student  users  were  not  trained  in  this  area. 

4.  Reasons  for  "kill  point"  activation  were  too 
vague . 

2.  Subjective  Rating  Data 

Subjective  rating  data  were  collected  using  the  Student  IDCTT 
Trainer  Survey,  Source-of -Workload  Evaluation,  and  User  Interface 
Dimensions  Questionnaire .  Each  survey  measure  solicited  different  aspects 
of  the  trainer  by  responses  on  a  scale  from  1  to  11.  The  scale  values  and 
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interquartile  ranges  (IQR)  for  these  surveys  are  presented  in  Tables  7,  8, 
and  9  . 


TABLE  7  :  SCALE  VALUES  AND  INTERQUARTILE  RANGES  OF  ITEMS  EVALUATING  STUDENT 
IMPRESSIONS  OF  NINE  DESIGN  ASPECTS  OF  THE  IDCTT  TRAINER 


Question 

Scale  Value 

IQR 

How  easy  was  the  system  to  operate? 
(Difficult . .Easy) 

8.72 

1.47 

Relevant  information  options  on 
touchscreen  monitor?  (None.. All) 

7.09 

3.00 

How  easily  touchscreen  allowed 
information  input?  (Dif f icult .. Easy) 

7.91 

4.55 

Was  the  scenario  too  difficult? 

(Difficult. . Easy) 

4.39 

1.72 

Was  the  scenario  easy  to  understand? 
(Confusing. .Clear) 

8.14 

2.27 

Was  the  scenario's  pace  too  slow? 

(Fast . .Slow) 

5.61 

1.34 

Was  the  scenario  realistic? 

(Unrealistic. .Realistic) 

8.21 

1.87 

Usefulness  as  a  DC  School  Training  aid? 
(Worthless. .Useful) 

9.61 

1.23 

Usefulness  as  a  shipboard  training  aid? 
(Worthless. .Useful) 

9.61 

1.55 

TABLE  8 :  SCALE  VALUES  AND  INTERQUARTILE  RANGES  OF  ITEMS  EVALUATING 
STUDENTS  IMPRESSION  OF  EIGHT  INTERACTIVE  COURSEWARE  INTERFACE  DIMENSIONS 


Statement 


Scale  Value 


IQR 


Ease  of  use  (Dif f icult . .Easy ) 


8.32 


2.03 


Navigation  through  ICW  (Dif f icult . .Easy) 


8.39 


2.03 


Cognitive  load  (Unmanageable . .Manageable) 


8.74 


1.74 


Mapping  (None .. Powerful ) 


8.89 


1.45 


Knowledge  of  space  compatibility 
(Incompatible. .Compatible) 


8.93 


1.88 


Information  presentation  (Obtuse . .Clear) 


9.92 


1.43 


Media  Integration 
(Uncoordinated. .Coordinated) 


9.77 


l.*4 


Overall  Functionality 
(Dysfunctional. .Highly  Functional) 


9.64 


1.98 


TABLE  9 :  SCALE  VALUES  AND  INTERQUARTILE  RANGES  OF  ITEMS  EVALUATING 
STUDENT  IMPRESSIONS  OF  SIX  HUMAN  WORKLOAD  DEMAND  DIMENSIONS 


Dimansion 

Scale  Value 

IQR 

Mental  Demand  (Low.. High} 

9.76 

l  .  84 

Physical  Demand  (Low.. High) 

3.50 

3 .77 

Temporal  Demand  (Low.. High) 

9.21 

1.90 

Performance  Demand  (Low.. High) 

8.86 

2.16 

Effort  Demand  (Low.. High) 

9.50 

1.79 

Frustration  (Low. .High) 

6.50 

4.75 

3.  Check-Off  Problem  Peatures 

Students  were  presented  with  seven  IDCTT  Trainer  features  and  asked  to 
indicate  which  feature,  if  any,  caused  them  problems.  The  number  of 
student  responses  and  the  percentage  of  students  who  identified  each  topic 
as  an  area  where  improvements  should  be  made  are  provided  in  Table  10. 


TABLE  10  :  NUMBER  AND  PERCENTAGE  OF  STUDENTS  WHO  IDENTIFIED  CHECK-OFF  SHEET 
ITEMS  AS  FFATURES  OF  THE  IDCTT  TRAINER  WHICH  COULD  BE  IMPROVED 


IDCTT  Trainer  Feature 

Number  of 
Students  Who 
Checked  Each 
Feature  out  of 
32  Students 

Percentage  of 
the  32 

Students  Who 
Checked  Each 
Feature 

Operating  touchscreen  monitor 

19 

59 

Understanding  audio  reports 

2 

06 

Locating  DC  Plate  information 

7 

21 

Speed/Volume  of  information 

12 

38 

DC  alarm  panel  display 

0 

00 

Firemain  panel  and  pump  operations 

6 

19 

Other  features  not  listed 

5 

16 

4.  Instructor  Evaluations 

Short  essay  and  narrative  descriptions,  as  well  as  rating  data 
were  collected  from  the  Damage  Control  School  instructors  using  the  IDCTT 
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Instructor  Survey.  A  summary  of  the  instructor  comments  answers  is 
provided  in  Table  11.  Table  12  provides  the  results  from  the  instructor 
rating  responses  and  interquartile  ranges  from  the  five  rating  categories 
solicited.  Appendix  B  contains  a  complete  listing  of  all  the  instructors' 
short  essay  and  narrative  descriptions. 


TABLE  11:  SUMMARY  OF  INSTRUCTOR  SHORT  ESSAY  AND  NARRATIVE  RESPONSES  TO 
QUESTIONS  FROM  THE  INSTRUCTOR  IDCTT  TRAINER  SURVEY  ON  THE  OPERATION  OF  THE 
IDCTT  TRAINER 


Question 

Comments 

What  aspects 
did  you  like 
about  the  IDCTT 
Trainer  for 
teaching  damage 
control 
problems? 

1.  Ability  of  the  IDCTT  Trainer  to  present 
identical  scenarios  to  each  DCA  student. 

2.  The  level  of  realism  that  the  IDCTT  Trainer 
induces . 

3.  Ability  to  objectively  critique  the  DCA's 
actions . 

4.  Reinforces  basic  and  advanced  damage  control 
concepts . 

What  problems 
did  you 

encounter  while 
using  the  IDCTT 
Trainer  as  an 
instructional 
aid? 

1.  The  program  did  not  run  real  time,  rather  it 
moved  more  like  a  video  game. 

2.  Touchscreen  monitor  slowed  student  responses. 

3.  Student  unfamiliarity  with  the  system. 

4.  Extensive  pre-brief  in  classroom  required  to 
fully  prepare  students  for  IDCTT. 

What  aspects  of 
the  IDCTT 

Trainer  would 
you  like 
changed? 

1.  Provide  more  detailed  feedback  from  the  repair 
lockers  in  response  to  incorrect  or  improper 
orders . 

2.  Change  the  method  of  input ing  orders  from  the 
touchscreen  to  voice  recognition. 

3 .  CSMC  should  have  its  own  repair  team  to  isolate 
and  correct  chill  water  problems. 

4 .  Increase  the  number  of  scenarios . 

What  benefits 
do  you  envision 
from  the  use  of 
the  IDCTT 

Trainer  at  the 
Damage  Control 
School ? 

1.  May  be  used  as  a  final  simulation  to  objectively 
determine  a  students'  ability  to  operate  under 
stressful  conditions. 

2 .  Better  preparing  DCA  students  for  the  pressures 
and  problems  associated  with  actual  damage 

control  scenarios  in  the  fleet. 
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TABLE  12 :  SCALE  VALUES  AND  INTERQUARTILE  RANGES  OF  ITEMS  EVALUATING 
INSTRUCTORS  IMPRESSION  OF  FIVE  ASPECTS  OF  THE  IDCTT  TRAINER 


Question 

Scale 

Value 

IQR 

How  easily  did  IDCTT  allow  you  to 
instruct  students?  (Dif f icult . .Easy) 

6.50 

3.00 

How  realistic  was  the  IDCTT  Trainer? 
(Artificial. .Realistic) 

5.50 

3.50 

Extent  you  would  like  to  see  IDCTT 
installed  permanently  at  the  DCA  School? 
(Very  Little.. Very  Much) 

6.50 

2.00 

Rate  the  students  reaction  to  t%  IDCTT 

Trainer?  (Negative .. Positive) 

7.50 

1.50 

How  beneficial  would  IDCTT  be  as  a 
shipboard  trainer? 

(Detrimental. .Beneficial) 

5.00 

1.50 

B.  IDCTT  VERSUS  DC  CENTRAL  TRAINER  PERFORMANCE  COMPARISON 

The  same  32  students  were  used  in  the  IDCTT  versus  DC  Central  Trainer 
performance  comparison  as  those  in  the  IDCTT  Trainer  performance 
evaluation.  The  data  consisted  of  graded  student  performance  and  rating 

data.  The  results  are  presented  below. 

1.  Graded  Student  Runs 

Each  of  the  32  student  participants  received  two  grades,  one  for 
their  performance  in  the  IDCTT  Trainer  and  the  other  for  the  DC  Central 
Trainer.  These  scores  provided  paired  scoring  data  which  was  analyzed 
using  the  Wilcoxon  Signed  Rank  Test.  The  Wilcoxon  Signed  Rank  Test 
revealed  that  the  population  relative  frequency  distribution  of  the  IDCTT 
Trainer  performance  scores  was  shifted  to  the  right  of  the  DC  Central 
trainer  performance  scores  (.025  <  P  <  .01).  Table  13  compares  summary 
statistics  for  the  two  sets  of  students'  scores,  while  Appendix  C  contains 
the  raw  data  scores  and  the  Wilcoxon  Signed  Rank  Test  results. 
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TABLE  13:  SUMMARY  STATISTICS  FOR  STUDENT  SCORES  RECEIVED  DURING  THE 
PERFORMANCE  EVALUATION  SESSIONS  FOR  THE  IDCTT  TRAINER  AND  THE  DC  CENTRAL 
TRAINER 


Statistical  Parameter 

IDCTT  Trainer 

DC  Central  Trainer 

Parameter 

Parameter 

Mean 

89.34 

36.53 

Median 

91.5 

86.5 

Standard  Deviation 

6.24 

5.01 

Skewness 

LO 

00 

O 

1 

-0.06 

Minimum  Score  Achieved 

72 

79 

Maximum  Score  Achieved 

99 

93 

2.  Rating  Data 

Scale  values  and  interquartile  ranges  were  compiled  from  data 
collected  by  the  Student  IDCTT  vs  DC  Central  Trainer  Comparison  Survey  and 
the  Scenario  Topics  Ranking  Survey.  Table  14  compares  the  scale  values 
and  interquartile  ranges.  Scale  values  span  an  11  point  bipolar  scale: 
low  scores  indicate  a  preference  for  the  IDCTT  Trainer;  high  scores 
indicate  a  preference  for  the  DC  Central  Trainer,  a  six  is  neutral. 

Table  15  shows  the  scale  values,  interquartile  ranges  and  rank 
order  for  each  fundamental  damage  control  topic .  These  data  were  obtained 
by  the  IDCTT  and  DC  Central  Trainers'  Scenario  Topics  Ranking  Survey.  It 
should  be  noted  that  unlike  previous  survey  scales  used  in  the  present 
study  which  employed  an  eleven  point  range,  the  Scenario  Tonics  Ranking 
Survey,  students'  responded  on  a  seven  point  scale.  The  higher  the  scale 
value  for  a  topic  the  more  important  the  topic  was  in  successfully 
completing  the  scenario  problem. 
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TABLE  14:  SCALE  VALUES  AND  INTERQUARTILE  RANGES  OF  ITEMS  COMPARING 
STUDENTS  IMPRESSIONS  OF  THE  IDCTT  TRAINER  AGAINST  THE  DC  CENTRAL  TRAINER 
COMPARISON  ACROSS  11  OPERATIONAL  AND  DESIGN  CHARACTERISTICS 


Question 

Scale 

Value 

IQR 

Simulated  the  shipboard  environment  more 
realistically 

4.04 

2.73 

Enabled  instructors  to  provide  complete 
post  scenario  debriefs 

4.06 

3.36 

Produced  the  greatest  level  of  stress 

2.50 

3.26 

Allowed  instructors  to  monitor  student's 
performance 

2.71 

2.50 

Prepared  the  student  for  actual  shipboard 
emergencies 

4.06 

4.29 

Updated  student  inputs  more  easily 

3.83 

6.00 

Provided  scenario  information  closely 
resembling  shipboard  methods 

4.50 

4.50 

Provided  most  effective  teaching 
environment  to  exercise  damage  control 
skills 

3.50 

3.67 

Promoted  greater  learning  in  time 
allotted 

2.38 

2.47 

Preferred  training  method 

1.39 

2.89 

Stimulated  the  student  to  perform 

2.86 

2.75 

TABLE  15:  SCALE  VALUES,  INTERQUARTILE  RANGES  AND  RANK  ORDERS  COMPARING  THE 
IMPORTANCE  OF  FUNDAMENTAL  DAMAGE  CONTROL  TOPICS  IN  COMPLETING  THE  IDCTT 
AND  DC  CENTRAL  TRAINERS'  SCENARIO  BATTLE  PROBLEM 


Measure 

Scale  Value 

IQR 

Rank 

IDCTT 

DC 

Central 

IDCTT 

DC 

Central 

IDCTT 

DC 

Central 

Communications 

4.06 

6.11 

3.00 

1.56 

11 

3 

Inform  chain  of 
command 

4.19 

5.33 

2.37 

1.46 

10 

9 

Set  Zebra 

5.85 

5.21 

1.92 

1.80 

6 

10 

Restore  Vital 
Systems 

6.78 

6.37. 

0.85 

1.34 

1 

1 

Manage  DC 

Central 

4.96 

5.34 

3.62 

2.37 

9 

8 

6.61 

6.07 

1.04 

1.27 

2 

4 

Manned  and  Ready 

3.97 

4.17 

3.24 

2.64 

12 

12 

Restore  Firemain 

6.08 

5.89 

1.42 

1.45 

3 

6 

Manage  Personnel 
casualties 

1.98 

2.69 

1.96 

2.59 

13 

13 

Locate  damage 

5.60 

5.60 

1.69 

1.61 

8 

7 

Prioritize 

damage 

5.61 

5.18 

3.50 

2.84 

7 

11 

Coordinate  fire 
fighting  teams 

5.98 

6.17 

1.91 

1.32 

4 

2 

Isolate 

explosive 

hazards 

5.98 

6.06 

2.62 

1.05 

5 

5 
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IV.  DATA  ANALYSIS  AND  DISCUSSION 

Data  collected  from  the  validation  test  group  allowed  two 
circumscribed  analyses:  an  analysis  of  the  IDCTT  Trainer  performance  alone 
and  a  comparative  analysis  of  the  IDCTT  versus  the  DC  Central  Trainer. 
This  chapter  analyzes  the  results  for  these  two  evaluations.  The  first 
part  addresses  the  IDCTT  Trainer  performance  evaluation  and  reports  the 
findings  from  the  short  essay,  rating  data,  check-off  problem  features, 
and  instructor  evaluations.  The  second  part  contrasts  the  IDCTT  Trainer's 
performance  with  the  DC  Central  trainer,  using  the  rating  data  and  short 
essay  responses  reported  in  the  last  chapter. 

A.  IDCTT  TRAINER  PERFORMANCE  EVALUATION 

The  IDCTT  Trainer  performance  evaluation's  objective  was  to  collect 
sufficient  data  to  determine  the  effectiveness  of  the  IDCTT  Trainer  as  a 
student  training  aid  and  an  instructor  teaching  tool.  The  focus  or  the 
present  treatment  is  four-fold.  First,  to  determine  if  users  easily 
interacted  within  the  IDCTT  environment  and  to  examine  features  of  the 
IDCTT  Trainer  which  might  be  improved.  Second,  the  scenario  itself  was 
examined  to  determine  its  effectiveness  as  a  simulated  battle  training 
problem.  T:  ,  the  actual  student  workload  using  the  IDCTT  Trainer  was 

evaluated  to  determine  if  the  trainer  provided  an  acceptable  training 
environment.  Finally,  the  instructors'  evaluation  of  the  IDCTT  Trainer 
was  examined  to  establish  a  basis  in  experienced  opinion  regarding  the 
system's  usefulness  as  a  training  tool.  These  four  themes  are  presented 
in  the  following  four  sections: 

•  System  User  Interface, 

•  Scenario  Critique, 

•  Student  Workload,  and 

•  Instructor  Evaluation. 
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1.  System  User  Interface 

The  Student  IDCTT  Trainer  Survey  and  the  User  Interface 
Dimensions  Survey,  were  used  to  determine  students'  reaction  to  the  IDCTT 
Trainer's  hardware  and  software  features.  These  surveys  provided  short 
essay  information,  rating  data,  and  check-off  problem  features  as 

discussed  in  Chapter  Two.  The  results  are  discussed  below. 

a.  Student  IDCTT  Trainer  Survey  Analysis 

Scale  values  calculated  from  the  Student  IDCTT  Trainer  Survey 
indicated  that  students  liked  the  IDCTT  Trainer  package  and  felt  it  was  an 
extremely  useful  training  aid  (see  Figure  2) .  Students  generally  agreed 
( IQR  =  1.47)  that  the  system  was  easy  to  operate  (S  =  8.72).  Moreover, 
respondents  indicated  a  slightly  stronger  than  neutral  attitude  that  the 
information  options  presented  by  the  Command  and  Control  Console  were 
adequate  to  successfully  combat  the  scenario  (S  =  7.09);  however,  their 
responses  to  this  question  tended  to  vary  more  widely  (IQR  =  3.00). 
Interestingly,  the  survey  revealed  that  students  generally  accepted  the 
touchscreen  monitor  as  an  easy  method  for  inputing  data  into  the  system  (S 
=  7.91),  despite  comments  in  the  essays  and  features  check-off  section 
which  identified  the  touchscreen  monitor  as  an  area  where  improvements 
should  be  made.30  Students'  responses  on  items  concerning  the  touchscreen 
monitor's  performance  revealed  a  substantial  lack  of  consensus  (IQR  of 
4.55)  . 

The  trainer's  ratings  of  usefulness  as  a  DC  School  training 
aid  or  as  a  potential  shipboard  training  aid,  were  very  high  and 
consistent  across  students.  The  usefulness  ratings  were  9.61  in  each 

case,  with  very  low  IQRs  of  1.23  and  1.55  respectively. 

b.  Operating  System 

Students  were  clearly  favorably  impressed  with  the  system's 
operational  characteristics.  Despite  the  high  positive  scale  values 


These  findings  will  be  discussed  later. 
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Student  1DCTT  Trainer  Survey 
System  Operations  Results 


Negtiive 

Responses 


Fodttve 


5  6  7  S  9  10  11 


How  Easy  the  System  is  to  Opacate 
(Difficult.. Easy) 

Relevant  Information  Options  On  the 
Touchscreen  Moodtv  (Nane..All) 

How  Easily  Touchscreen  Allowed 
Information  Input  (Difficult.  .Easy) 

Usefulness  as  a  DC  School  Training 
Aid  (Worthless.  .Useful) 

Usefulness  as  a  Shipboard  Training 
Aid  (W arthless .  .Useful) 


b-sH 
1 — *— I 


Figure  2:  Scale  Values  and  Interquartile  Ranges  of  Items  Evaluating 
Student  Impressions  of  Five  Design  Aspects  of  the  IDCTT  Trainer 

associated  with  the  touchscreen  monitor's  operational  characteristics, 
most  of  the  problems  identified  with  the  system  were  rooted  in  using  the 
touchscreen  monitor  to  input  information  into  the  system.  The  Check-Off 
Problem  Features  List  was  administered  to  identify  areas  in  the  IDCTT 
Trainer  which  needed  improvement.  Figure  3,  graphically  displays  the 
percentage  of  students  who  identified  each  of  the  seven  IDCTT  Trainer 
features  as  areas  where  they  encountered  difficulties.31  The  following 
discusses  these  findings  as  they  relate  to  seven  specific  features: 

•  Operating  the  touchscreen  monitor, 

•  Understanding  audio  reports, 

•  Locating  DC  Plate  information, 


31  The  seventh  IDCTT  Trainer  feature  was  an  “other"  category,  enabling 
students  to  identify  problem  areas  not  listed. 
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•  Speed  or  volume  of  information  present?1, 

•  DC  Alarm  Panel  display, 

•  Firemain  Panel  and  firepump  operations,  and 

•  Other  features  not  listed. 


Check-Off  Problem  Features 


Percentage  of  Student  Responses 


— i — — i — 1 — i — — i — * — i — * — i — * — i — 

Touchscreen  Audc  DC  Plato  Pace  DC  Panto  Fire  Panto  Other 


Figure  3:  percent  of  Students  Indicating  Problems  with  Seven  IDCTT 
Trainer  Features 

(1)  Touchscreen  Monitor.  Fifty-nine  percent  of  the  students 
indicated  they  experienced  some  difficulty  operating  the  touchscreen 
monitor.  More  specifically,  they  identified  difficulties  inputing 
information  into  the  touchscreen  as  the  cause  for  missing  key  audio 
reports,  and  in  seme  cases,  losing  control  of  the  battle  problem.  The 
touchscreen's  primary  problem  was  its  low  sensitivity  to  touch  and  its 
slow  response  time.  Students  often  selected  the  correct  button,  but  the 
touchscreen  did  not  sense  their  input.  Further,  the  actual  physical  I 

distance  between  the  buttons  was  narrow,  causing  students  to  accidentally 

I 

I 

I 

I 
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choose  the  wrong  option.  This  was  particularly  frustrating  when  input ing 
fire  and  flooding  boundaries.  Ordering  boundaries  required  pressing  a 
sequence  of  up  to  14  buttons,  taking  upwards  of  30  seconds  of  problem 
clock  time  to  complete.  An  accidental  input  during  this  sequence  could 
seriously  affect  the  student's  performance  by  causing  them  to  focus  their 
attention  on  repeating  the  14  button  process  rather  than  the  flow  of 
information  from  the  various  sources.  Most  of  the  students  opted  to  use 
the  computer  keyboard  instead  of  the  touchscreen's  numeric  key  pad. 
However,  switching  back  and  forth  between  the  computer  keyboard  and  the 
touchscreen  monitor  was  difficult  because  only  numeric  inputs  and  the 
enter  command  could  be  initiated  from  the  computer  keyboard. 

Many  of  the  students  had  never  used  a  touchscreen 
monitor  before  this  study,  but  they  became  markedly  faster  with  practice. 
Still,  comments  in  student  essays  revealed  an  underlying  sentiment  to 
either  increase  the  sensitivity  of  the  touchscreen  or  explore  different 
methods  for  input ing  information.  These  suggestions  included: 

•  Voice  activation, 

•  Using  a  mouse, 

•  Making  all  action  available  on  the  computer  keyboard,  and 

•  Designating  an  individual  to  input  all  orders  into  the  touchscreen. 

(2)  Volume  of  Information .  The  next  most  frequently 

reported  problem  concerned  the  speed  or  volume  of  information  presented. 
Thirty-eight  percent  of  the  students  indicated  they  had  problems  with  the 
pace  of  information  flow,  but  student  essay  responses  did  not  provide 
information  as  to  why:  only  five  students  specifically  addressed  this 
area.  Although  some  students  seemed  to  be  overwhelmed  with  the  speed  and 
volume  of  information  during  their  practice  run,  most  appeared  to  be 
comfortable  with  the  pace  by  the  time  they  reached  their  graded  session. 
Some  students  specifically  identified  the  fast  pace  as  instrumental  in 
making  the  trainer  scenario  seem  more  realistic .  Interestingly,  one 
student  suggested  programming  different  difficulty  levels,  such  as 
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“beginner",  “intermediate",  and  “advanced",  thus  enabling  slower  students 

to  practice  at  levels  more  conducive  to  their  stage  of  skill  development. 

(3)  DC  Plate,  Firemain  Alarm  Panel  and  Pump  Operations . 

Approximately  the  same  percentage  of  respondents  identified  “locating  DC 
Plate  information"  (21  percent),  and  the  “Firemain  Alarm  Panel  and  pump 
operations"  (19  percent),  as  problem  areas  during  their  evaluation  trials. 
Students  attributed  their  difficulties  locating  DC  Plate  information  to 
their  unfamiliarity  with  the  DDG-51  class.  Although  this  is  not  a 
shortcoming  of  the  IDCTT  Trainer,  it  indicates  the  need  for  more  scenarios 
with  different  ship  classes  to  support  the  diversity  of  fleet  experience. 
While  the  IDCTT  Trainer  was  not  designed  to  instruct  students  on  reading 
DC  Plates,  its  continued  use  will  familiarize  students  with  the  DDG-51 
class  ship  generally  and  reading  their  DC  Plates  specifically. 

Respondents  indicated  the  main  problem  with  the  Piremain 
Alarm  Panel  was  the  small  display  screen.  The  Firemain  Panel  is  displayed 
on  a  15  inch  color  monitor.  Since  the  display  screen  symbolically 
represents  over  thirty  of  the  primary  firemain  valves  and  six  firepumps, 
the  size  of  the  valves  and  pumps  are  small  given  the  limited  screen  space. 
Students  often  overlooked  critical  Indications  on  the  Piremain  Panel 
because  they  simply  did  not  notice  any  changes  in  the  compact  display. 
Two  solutions  to  this  problem  are  readily  apparent. 

•  A  larger  Firemain  Panel  Screen  would  proportionally  increase  the  size 
of  each  Firemain  Panel  feature  without  altering  the  present  display, 
and 

•  Reducing  the  number  of  valves  displayed  to  those  critical  to  the 
scenario,  thus  allowing  more  room  for  those  remaining. 

Adopting  a  larger  screen  is  preferred  because  it  enables  the  students  to 

actually  see  a  representation  of  the  DDG-51  class  Firemain  Panel. 

(4)  Audio  Reports,  DC  Alarm  Panel,  and  Features  Not  Listed. 

The  areas  which  needed  little  or  no  improvements  were,  “other  features  not 
listed  in  the  survey"  (16  percent),  audio  reports  (6  percent),  and  the  DC 
alarm  panel  (no  responses) .  Un familiarity  with  the  system,  particularly 
not  knowing  what  actions  the  IDCTT  Trainer  did  automatically  and  what 
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orders  had  to  be  manually  inputed  to  accomplish  specific  tasks,  summarize 
the  responses  for  the  features  not  listed  in  the  survey.  Students 
suggested  using  a  demonstration  tape  to  provide  a  step  by  step  sequence 
through  the  various  opt  ions. 3- 

For  audio  reports,  respondents  indicated  that  they 
became  distracted  by  erroneous  information  in  the  background  noise. 
Background  noise  is  simply  a  compilation  of  actual  reports  delivered  to 
the  DCA  over  the  Command  and  Control  Console.  It  is  played  continuously 
on  a  separate  audio  track.  Unfortunately,  many  of  these  reports  are 
erroneous,33  depending  on  the  student's  previous  corrective  actions  to 
combat  the  damage.  This  problem  could  be  easily  rectified  by  recording 
background  information  that  is  relevant,  but  general  in  nature.  This 
would  provide  an  appropriate  distraction,  but  would  not  give  the  DCA  mis¬ 
information  . 

No  student  in  the  study  identified  any  problems  with  the 
DC  Alarm  Panel.  Although  the  DC  Alarm  Panel  was  displayed  on  a  15  inch 
color  monitor  similar  to  the  Firemain  Alarm  Panel,  there  were  no  problems 
reported  with  the  size  of  the  graphic  representation  of  the  various 
alarms . 


3:  Each  student  attended  a  90  minute  lecture  on  the  background  and  functions 
of  the  IDCTT  Trainer.  This  lecture  used  overhead  transparencies  of  the  IDCTT 
Trainer's  screen  displays  rather  than  the  actual  system.  Respondents  felt  the 
overhead  transparencies  did  not  sufficiently  prepare  the  students  for  the  actual 
trainer . 

Although  there  is  a  demonstration  option  on  the  IDCTT  Touchscreen,  this  option 
simply  repeats  a  complete  scenario,  darkening  the  buttons  that  would  be  pressed 
for  the  various  actions  taking  place  on  the  screen.  There  is  no  audio  voice  over 
in  the  demo  to  explain  the  actions  that  are  being  taken.  Further,  watching  the 
present  demo  tape  prior  to  system  use  would  predisclose  the  battle  problem 
because  it  is  the  same  scenario  that  the  student  would  use. 

"  Erroneous  reports  include,  "High  Temperature  Alarm  in  the  VLS  Magazine", 
"Loss  of  Firemain  pressure",  and  "Complete  loss  of  Chill  Water,  no  radar  arrays 
on  line” .  Each  of  these  reports  potentially  cause  the  student  to  take  unnecessary 
corrective  actions. 
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c.  u ser  Interface  Dimensions 


The  User  Interface  Dimensions  Survey  measured  student 
responses  across  eight  conventional  interactive  courseware  dimensions. 
Student  responses  were  clearly  positive  on  each  dimension;  the  scale 
values  ranged  from  8.32  to  9.92.  Further,  there  was  little  variation  in 
the  student  responses;  the  IQR  ranged  from  1.43  to  2. 03. 34  Figure  4 
depicts  these  results  graphically. 


User  Interface  Dimensions 
Questionnaire  Results 

Negative 

Beqwwes 

1  2  3  4  5  6 

1  1  1  1  1  1 

^odttve 

7  t  9  10  11 

1  1  1  1  1 

Ease  of  Use  (Difficult .Easy) 

1  1  1  1  1  1 

1  1  1  1  1 

1 - * - 1 

Navigation  Through  ICW 

(I>ifficult..Easy) 

) - It - 1 

Cognitive  Load 
(Unmanageable.  .Manageable) 

1 * — 1 

Mapping  (Nooe  JWerful) 

I — *— t 

Knowledge  of  Space  Compatibility 
(Incampatfbto.  .Compatible) 

| - * - | 

Himiuiiiuii  riwuumi 
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1 — * — 1 

Media  Integration 
(Uncoordinated  .Coordinated) 

1 - * - 1 

Overall  Functionality 
(DysfuctkmaL  .Highly  Functional) 

1 - * - 1 

Figure  4:  Scale  Values  and  Interquartile  Ranges  of  Items  Evaluating 


Student  Impressions  on  Eight  ICW  Interface  Dimensions 

2.  Scenario  Critique 

Figure  5  reveals  that  student  reactions  to  the  IDCTT  Trainer's 
scenario  were  highly  positive.  The  validation  group  reported  that  the 

scenario  was  easy  to  understand  (S  =  8.14)  and  was  extremely  realistic  (S 


34  See  Table  6  for  a  complete  summary  of  the  scale  values  and  interquartile 
ranges  for  each  interface  dimension. 
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=  8.21).  Responses  on  both  aspects  of  the  scenario  showed  little 
variation  as  reflected  in  the  narrow  IQRs  of  1.87  and  2.27,  respectively. 
Students  were  neutral  and  in  agreement  on  the  pace  of  the  scenario  (S  = 
5.60;  IQR  of  1.34).  Finally,  the  validation  group  agreed  that  the 
scenario  was  difficult  to  complete  (S  =  4.39;  IQR  =  1.72)  .  The  following 
four  sections  address  various  aspects  of  the  scenario  used  and  student 
reactions  to  it. 


Student  IDCTT  Trainer 

Scenario  Question  Responses 

Nejudve 

Positive 

Responses 

Rwpooses 

1  2  3  4  S  6 

1  1  1  1  1  1 
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Was  the  Scenario  Easy  to  Undantaod 
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1 — « - 1 

W |K  the  I fm  Slaw  i  V  j 

(Fast-Slow) 

Was  the  Scenario  Realistic 
(Umeafisric-Raalistic) 

1 - * - 1 

Figure  5:  Scale  Values  and  Interquartile  Ranges  of  Items  Evaluating 
Student  Impressions  of  Four  Scenario  Design  Aspects 

a.  Scenario  Realism 

Student  narratives  cited  audio  and  visual  effects  as  the  main 
reasons  why  the  trainer  was  so  successful  in  creating  what  they  considered 

a  realistic  training  scenario.  Further,  they  revealed  that  the  realism 
was  heightened  in  that  the  printed  damage  control  chits  provided  another 
information  source  they  would  receive  if  they  were  performing  in  an  actual 
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shipboard  battle  damage  situation.  Moreover,  they  reported  that  the 
information  flow  closely  tracked  that  which  they  wou.d  experience  in  an 

actual  shipboard  multi-hit  scenario. 

Jb.  Scenario  Clarity 

Essay  responses  and  narra  ives,  clearly  indicated  that  the 
scenario  objectives  were  clearly  defined .  Student  responses  repeatedly 
suggested  that  the  progression  of  steps  needed  to  complete  the  scenario 
tracked  the  standard  procedures  taught  at  the  Damage  Control  School.35 
Two  items,  however,  were  identified  as  potential  problems  with 
understanding  the  IDCTT  Trainer  scenario.  The  first  was  an  insufficient 
knowledge  of  the  chill  water  system  to  make  rational  decisions  on  the 
corrective  actions  necessary  to  isolate  and  repair  it.  The  second  problem 
stemmed  from  an  underlying  unfamiliarity  with  the  DDG-S1  Class  Destroyer 
which  made  it  difficult  for  students  to  fully  comprehend  the  information 
they  received.  These  two  topics  are  amplified. 

(1)  Chill  Water.  Some  of  the  students  identified  ignorance 

of  chill  water  system  as  the  basis  for  their  inability  to  successfully 
complete  the  scenario.  Fourteen  of  the  32  students  (44  percent)  did  not 
complete  the  scenario  due  to  chill  water  related  problems.  The  IDCTT 
Trainer  scenario  emphasized  chill  water  system  restoration  as  one  of  the 
primary  actions  needed  to  complete  the  problem,  and  did  so  to  attain  the 
central  goal  of  incorporating  TSS  concepts  into  the  battle  problem.  The 
chill  water  system,  however,  was  not  taught  at  the  Damage  Control  School 
and  students  were  left  with  the  limited  knowledge  they  obtained  at 
previous  schools  or  experienced  in  the  fleet.  The  Damage  Control  School 
should  review  its  course  curriculum  and  place  a  stronger  emphasis  on  the 
chill  water  system  if  the  curriculum  is  to  reflect  the  shift  in  fleet 
doctrine  toward  the  TSS  philosophy. 


''  Successful  completion  of  the  scenario  requires  students  to  investigate 
damage,  set  boundaries,  combat  damage,  and  repair  the  damage.  This  broad 
sequence  of  events  are  the  fundamental  steps  taught  to  damage  control  students 
in  managing  damage  control  situations. 
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(2)  DDG-51  Class  Unfamiliarity .  Another  scenario  related 
problem  was  that  the  scenario  was  based  on  a  class  of  ship  with  which  most 
were  unfamiliar.  Since  the  DDG-51  Class  is  the  newest  ship  class,  most  of 
the  students  were  unfamiliar  with  its  design,  making  it  difficult  to 
locate  compartments  and  valve  numbers  from  the  DC  Plates.  Although  the 
students  were  not  familiar  with  the  DDG-51  Class  design,  the  concepts  used 
to  combat  damage  are  the  same,  regardless  of  the  ship's  design,  and 
students  who  employed  solid  damage  control  concepts  were  able  to 
successfully  complete  the  scenario,  even  if  they  did  not  know  the  ship 
class  specifics.  When  scenarios  are  written  for  different  ship  classes, 
students  will  be  able  to  choose  their  ship  class  from  a  library  of 
scenarios,  tailoring  training  to  the  knowledge  they  need  to  bring  back  to 
the  fleet. 

c.  Scenario  Pace 

Students'  reaction  to  the  scenario  pace  was  neutral. 
Although  some  students  fell  behind  in  the  problem,  they  attributed  the  lag 
to  the  slow  process  of  inputing  information  through  the  touchscreen  rather 
than  the  flow  of  information  itself.  Most  students  felt  that  the  fast 
flow  of  information  added  to  the  realism  of  the  scenario  and  induced  a 
level  of  stress  that  would  be  present  under  an  actual  damage  control 
scenario . 

d.  Scenario  Difficulty  Level 

Students  felt  the  scenario  was  difficult  and  only  one  out  of 
32  students  successfully  completed  the  scenario  during  the  practice 
session.  This  low  completion  was  partially  attributed  to  the  students' 
unfamiliarity  with  the  system's  operating  procedures  during  the  practice 
vun.  However,  the  low  completion  rate  underscores  the  difficulty  of  the 
scenario.  The  scenario  was  not  designed  to  be  easy,  but  to  provide  a 
challenging  learning  environment  in  which  students  could  practice  the 
damage  control  concepts  they  learned  in  the  classroom.  Further,  it  was 
designed  to  identify  student  mistakes  so  this  knowledge  could  be  applied 
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and  those  mistakes  rectified  in  subsequent  trainer  sessions.  Thus,  the 

students '  preveiling  experiences  the t  the  scenerio  wee  difficult  reflected 

the  designers  originel  intentions :  the  exercise  wes  to  be  difficult. 

3 .  Student  Workload 

Student  workload  was  evaluated  using  NASA's  Source  of  Workload 
Evaluation.  The  IDCTT  Trainer  was  developed  to  produce  a  learning 
environment  that  imposed  significant  mental,  temporal,  and  attentional 
demands.3'  .Respondents  rated  the  system  es  mantel  ly  end  temporelly 
demending  while  requiring  a  high  lev.  t  to  complete  the  IDCTT 

Trainer  scenario.  Scale  values  for  these  categories  ranged  from  9.21  to 
9.76,  with  very  narrow  ranges  (IQRs  of  1.79  and  1.90,  respectively).  As 
one  would  expect  from  a  computer  based  simulation,  students  rated  the 
physical  demand  required  by  the  system  as  low  (S  =  3.50).  When  asked  to 
rate  the  level  of  frustration  they  encountered  while  interacting  with  the 
system,  student  responses  were  essentially  neutral  (S  =  6.50),  but 

reflected  significantly  more  variation  in  that  assessment  ( IQR  =  4.75). 

A  review  of  students'  performance  grades  indicated  that  those  who 
performed  poorly  were  more  likely  to  be  frustrated  while  using  the  system 
than  those  who  performed  well  or  vice  versa.  This  finding  explains  the 
wide  range  of  responses  to  the  "level  of  frustration"  survey  question. 
Most  students  were  satisfied  with  their  personal  performance  level  (S  = 
8.86;  IQR  =  2.36) .  Figure  6  graphically  depicts  the  student  responses  for 

the  various  categories. 

4.  Instructor  Evaluation 

The  instructors'  evaluations  of  the  IDCTT  Trainer  were  generally 
neutral.  Figure  7  shows  the  instructors'  responses.  Unfortunately,  only 
three  of  the  seven  instructors  at  the  DCA  school  (43  percent)  spent  more 
than  two  hours  operating  the  system  and  assisting  students .  The  remaining 


■'  See  the  Subject  Instructions  and  Survey  for  the  Sources  of  Workload 
Evaluation  contained  in  Appendix  B  for  a  complete  description  of  each  workload 
factor . 
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four  instructors  formed  their  opinion  on  the  system  by: 

•  Personally  completing  two  scenario  runs,  and 

•  Observing  the  other  instructors  assisting  students  during  their 
practice  and  test  sessions. 

This  could  account  for  their  neutral  sentiments  for: 

•  How  easily  the  IDCTT  allowed  them  to  instruct  students, 

•  How  realistic  the  IDCTT  Trainer  was,  and 

•  The  extent  they  would  have  liked  to  see  the  IDCTT  Trainer  permanently 
installed  at  the  DCA  School. 

Instructor  interviews  revealed  that  the  reason  for  the  neutral  response 
concerning  the  permanent  installation  of  the  IDCTT  Trainer  at  the  DCA 
School  was  that  a  clear  mission  in  the  school's  curriculum  was  not  clearly 
defined .  Since  the  system  was  not  formally  programmed  to  replace  the  DC 
Central  Trainer,  some  instructors  were  not  supportive  of  a  separate  IDCTT 


5S 


Trainer  requirement.  They  did  acknowledge,  however,  the  stu dents' 
positive  acceptance  of  the  system  (S  =  7.50;  IQR  =  1.50). 


Figure  7:  Scale  Values  and  Interquartile  Ranges  of  Items  Evaluating 
Instructor  Impressions  of  Five  IDCTT  Trainer  Design  Features 


B.  IDCTT  VERSUS  DC  CENTRAL  TRAINER  PERFORMANCE  COMPARISON 

The  DC  Central  Trainer  provided  a  clear-cut  baseline  against  which  to 
compare  the  IDCTT  Trainer.  Since  students  were  exposed  to  each  system  in 
a  24  hour  period,  they  responded  to  the  IDCTT  Trainer  vs  DC  Central 
Comparison  Survey  with  recent  experience  in  using  each  system.  Further, 
ranking  the  importance  of  various  damage  control  fundamentals  using  the 
Scenario  Topics  Ranking  Survey,  provided  a  measure  for  assessing  the 
differing  importance  they  attributed  to  13  fundamental  damage  control 
topics  for  each  trainer.  Once  the  level  of  importance  for  each  of  these 
topics  was  established,  the  topics'  relative  importance  for  each  of  the 
two  trainers  was  compared  to  determine  which  method  provided  a  more 
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standardized  scenario  Finally,  the  performance  grades  provided  a  simple 
way  to  evaluate  student  effectiveness  in  using  the  two  trainers  to  combat 

the  same  battle  problem.  These  comparisons  are  discussed  below. 

1.  Student  Trainer  Comparison 

Results  from  each  category  on  the  Comparison  Survey  clearly 
indicated  that  students  preferred  the  IDCTT  Trainer  over  the  present  DC 
Central  Trainer.  The  most  emphatic  response  from  this  comparative  survey 
was  the  computed  scale  value  of  1.39  which  indicated  that  the  IDCTT 
Trainer  is,  by  far,  the  preferred  training  method.37  Respondents  also 
strongly  indicated  that  the  IDCTT  Trainer  promoted  greater  learning, 
produced  significantly  more  stress,  and  stimulated  them  to  learn  much  more 
them  the  DC  Central  Trainer  did  (S  =  2.38,  2.50,  and  2.38;  IQR  =  2.47, 
3.26,  and  2.47,  respectively).  Many  students  described  a  greater  sense  of 
accomplishment  after  completing  the  IDCTT  Trainer,  since  it  seemed  more 
challenging.  Although  the  instructors  felt  that  the  IDCTT  Trainer  did  not 
significantly  increase  their  ability  to  evaluate  student  performance,  the 
students  themselves  strongly  felt  that  the  IDCTT  Trainer  provided  the 
instructor  with  a  better  method  to  assess  their  problem  solving 
performance  (S  =  2.71;  IQR  =  2.50). 

Responses  on  three  of  the  comparison  survey  questions  yielded  no 
consensus  among  student  responses.  These  questions  were: 

•  Method  which  better  prepares  students  for  shipboard  casualties 

(S  =  4.06;  IQR  of  4.29) , 

•  Easier  method  to  initiate  actions  (S  =  3.83;  IQR  of  6.00),  and 

•  Method  with  the  most  realistic  information  presentation  (S  =  4.50; 

IQR  of  4.50). 

These  three  items,  and  their  attendant  variability,  appeared  to  be  based 
two  schools  of  thought  on  how  actions  should  be  initiated,  and  information 
transferred  in  simulated  battle  problem  training.  One  group  of  students 
preferred  the  Command  Console's  simulated  shipboard  environment  and  push 

-  Low  values  indicate  a  preference  for  the  IDCTT  Trainer  while  High  values 
indicate  a  preference  for  the  DC  Central  Trainer. 
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button  displays.  The  other  group  felt  that  the  interaction  among 
watchstanders  was  a  key  portion  of  the  training  experience  and  preferred 
using  numerous  Sound  Powered  Phone  Talkers  to  relay  and  receive 
information.  The  critical  design  feature  upon  which  these  responses 
depended,  however,  was  how  much  difficulty  students  had  when  they  used  the 
touchscreen  monitor.  Those  who  experienced  difficulties  using  the 
touchscreen,  preferred  the  DC  Central  Trainer  method  of  initiating  actions 
through  Sound  Powered  Phone  Talkers .  Students  who  did  not  experience 
problems  with  the  touchscreen,  preferred  the  IDCTT  Trainer's  method. 
Preferences  were  clearly  related  to  the  users  mastery  of  the  touchscreen 
features  Figure  8  graphically  displays  the  student  responses  across  the 
11  comparative  dimensions. 


Student  IDCTT  vs  DC  Central 
Trainer  Comparison  Survey 
Results 


Realistic  Environment 

Rjst  Scenario  Debriefs 

Greatest  Level  of  Stress 

Instructor  Monitoring 

Prepares  Student  for  Actual 
Casualties 

Initiate  Damage  Control  Actions 
Realistic  Infonnatian.  Presentation 
Mc*e  Effective  Teaching  Medium 
Promotes  Greater  Learning 
Preferred  Training  Method 
Stimulated  You  to  Perform 


IDCTT 
1  2 


DC  Genual 
10  11 


-X- 


Figure  8:  Scale  Values  and  Interquartile  Ranges  of  Items  Comparing 
Student  Impressions  of  11  Trainer  Design  Features 
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2. 


Student  Performance  Grades 


Student  performance  grades,  collected  from  the  two  training 

methods,  were  used  to  determine  which  method  provided  a  better  environment 

in  which  students  performed.  As  discussed  previously,  the  scenario 

difficulty  level  was  judged  to  be  the  same  and  students  were  graded  using 

the  same  grading  criteria  for  each  method.38  The  Wilcoxon  Signed-Rank 

Test  for  a  Paired  Experiment  was  used  to  determine  if  the  relative 

frequency  distribution  of  student  performance  scores  were  identical,  or  if 

the  IDCTT  Trainer  scores  were  higher  than  the  DC  Central  Trainer  scores. 

The  results  of  this  test  identified  which  medium  promoted  higher  student 

performance.  Table  16  summarizes  the  results  of  this  test  while  a 

complete  set  of  calculations  is  included  in  Appendix  C. 

TABLE  16:  WILCOXON  SIGNED-RANK  TEST  SUMMARY  FOR  STUDENT  PERFORMANCE 
SCORES  FROM  THE  IDCTT  AND  DC  CENTRAL  TRAINERS 


Null  Hypothesis:  H0:  The  population  distributions  for  the  IDCTT 
Trainer  and  the  DC  Central  Trainer  performance  scores  are 
identical . 

Alternative  Hypothesis:  Ha:  The  population  relative  frequency 
distribution  of  the  IDCTT  Trainer's  performance  scores  is 
shifted  to  the  right  of  the  DC  Central  Trainer's  performance 
scores . 

Test  Statistic:  z  =  1.99 

Rejection  Region:  Reject  H0  if  z  >  za 

where  zn  =  1.96  at  the  .025  significance  level 

Conclusion:  Reject  Hc  and  accept  Ha  at  the  .025  significance  level. 


The  Wilcoxon  Signed-Rank  Test  indicated  that  the  frequency 
distribution  of  scores  received  using  the  IDCTT  Trainer  were  higher  than 
those  scores  received  using  the  DC  Central  Trainer  (.025  <  P  <  .01).  The 

higher  scores  obtained  from  the  IDCTT  Trainer  can  be  attributed  at  least 
three  factors  which  are  discussed  below. 

3s  Appendix  A  provides  a  copy  of  the  student  grading  criteria  used  in  the 
study . 
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a.  Student  Preference  for  the  IDCTT  Trainer 


That  students  preferred  the  IDCTT  Trainer  to  the  DC  Central 
Trainer  was  clearly  reflected  by  the  scale  values  from  the  Comparison 
Survey.  In  particular,  they  held  that  the  IDCTT  Trainer  promoted  greater 
learning,  and  stimulated  them  to  perform  to  the  best  of  their  ability. 
Higher  test  scores.,  therefore,  could  be  accounted  for  by  students  who  were 
simply  more  motivated  to  perform  in  the  IDCTT  Trainer.  In  their 
narratives  and  verbal  remarks  after  the  evaluation  period,  students 
indicated  that  the  clear  flow  of  information  associated  with  the  IDCTT 
Trainer  was  critical  to  their  success.  Conversely,  in  the  DC  Central 
Trainer,  students  reported  that  they  were  dependent  on  the  abilities  of 
their  Phone  Talkers  to  relay  the  correct  information,  a  dependency  which 

sometimes  hindered  the  timely  flow  of  accurate  information. 

b.  System  Stress 

The  IDCTT  Trainer  was  designed  to  induce  significant  amounts 
of  stress  on  students  through  the  rapid  pace  of  the  scenario  and  the 
content  of  various  audio  and  visual  stimuli.  The  subject  of  performance 
under  stress  is  not  the  focus  of  this  paper,  but  it  was  readily  observed 
that  students  who  were  exposed  to  the  stressful  IDCTT  scenario  appeared  to 
concentrate  more  intently,  and  try  harder  to  successfully  complete  the 
battle  problem  than  when  operating  in  the  less  stressful  DC  Central 
Trainer  environment.  When  in  the  less  stressful  DC  Central  Trainer, 
participants  projected  a  lax  attitude  toward  completing  the  battle 
problem,  and  in  general,  did  not  seem  to  take  the  scenario  as  seriously  as 
they  did  while  executing  the  IDCTT  scenario.  The  degree  to  which  students 
became  engrossed  in  the  scenario  seemed  to  correspond  with  their 

performance  scores. 

c.  Grading  Expectations 

Before  the  test  period,  performance  scores  were  expected  to 
be  higher  in  the  DC  Central  Trainer  due  to  students'  familiarity  with  the 
system  and  the  concerted  problem  solving  methods  the  group  trainer 
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allowed.  Two  factors  discounted  this  expectation.  First,  students 
quickly  became  familiar  with  the  IDCTT  Trainer.  Most  students  adeptly 
operated  the  IDCTT  Trainer  by  the  end  of  the  practice  session,  and  by  the 
graded  session,  each  student  completed  all  required  actions  without 
outside  assistance.  The  speed  with  which  students  adapted  to  the  computer 
simulation  quickly  eroded  any  advantage  the  DC  Central  Trainer  conferred 
because  of  its  familiarity.  Second,  when  students  were  removed  from  the 
group  atmosphere  of  the  DC  Central  Trainer,  they  assumed  greater 
responsibility  in  solving  the  problem  and  decision  making  became  more 
spontaneous.  The  quality  of  decisions  did  not  significantly  differ 
between  the  two  systems;  per  se,  however,  the  speed  with  which  decisions 
were  made  was  much  quicker  in  the  IDCTT  Trainer.  Thus,  the  DCA's 
performance  did  not  seem  to  be  hindered  by  not  having  a  watchteam  to 

suggest  various  damage  control  actions. 

d.  Summary 

Based  on  student  performance,  the  IDCTT  Trainer  is  a  more 
effective  training  medium.  Students  were  more  motivated  to  perform,  and 
preferred  the  methods  used  by  the  IDCTT  Trainer  to  deliver  simulated 
battle  problems.  The  IDCTT  Trainer  allowed  the  instructor  to  carefully 
monitor  student  performance  by  eliminating  the  need  for  the  instructor  to 
spend  much  of  his  time  initiating  the  scenario  rather  than  critiquing  it. 
Unlike  the  DC  Central  Trainer,  where  the  instructor  is  removed  from  the 
student  training  area,  the  IDCTT  instructor  is  positioned  to  critique  and 

assist  the  student  in  the  training  area. 

3.  Trainer  Type  Scenario  Variation 

Developing  a  procedure  to  measure  the  level  of  variation  induced 
by  each  method  over  many  repetitions  of  the  same  scenario  was  difficult. 
The  objective  of  this  portion  of  the  study  was  to  measure  the  extent  to 
which  each  trainer  consistently  delivered  a  standardized  scenario. 
Clearly,  reducing  the  level  of  variation  between  repetitions  so  each 
student  is  given  the  same  difficulty  level  in  which  to  train  is  desirable. 
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Likewise,  reducing  the  subjectivity  in  assigning  performance  scores 
between  training  periods  with  varying  levels  of  difficulty  is  desirable 
from  a  methological  perspective.  That  students  would  feel  the  computer 
simulation  would  provide  more  consistent  training  scenarios  was  expected 
because  the  computer  generated  scenario  did  not  rely  exclusively  on  the 
instructor's  ability  to  present  the  scenario,  as  is  the  case  with  the  DC 
Central  Trainer.  The  IDCTT  Trainer  is  physically  programed  to  execute  a 
series  of  events  which  presents  the  same  scenario  each  time.39 
Alternately,  the  DC  Central  Trainer's  scenario,  which  relies  on  the 
instructor  to  create  the  training  environment,  differs  between  trials  for 
four  important  reasons: 

•  Instructors  do  not  follow  a  specific  time-line  when  initiating  the 
scenario, 

•  Instructors  are  given  a  script  of  events  but  are  free  to  execute  each 
event  in  his  own  training  style, 

•  Instructors  often  adjust  the  scenario  difficulty  level  to  meet 
student  performance  level,  and 

•  Students  can  request  additional  information  from  the  instructors, 
which  is  not  an  option  with  the  IDCTT  Trainer. 

The  measures  used  to  analyze  the  level  of  variation  present  in  each 

trainer  type  were  the  scale  values  and  interquartile  ranges  calculated 

from  responses  on  the  Scenario  Topics  Ranking  Survey.  These  two  measures 

are  more  fully  discussed  below. 

a.  Scale  Value  Assessment 

The  numerical  index  used  to  evaluate  the  variability  in  the 
two  methods  reflected  the  degree  each  method  emphasized  the  various 
fundamental  damage  control  actions  required  to  complete  the  scenario.  The 
Scenario  Topics  Ranking  Survey  required  students  to  rate  how  important 
each  of  13  fundamental  damage  control  topics  was  in  completing  the 
scenario.  The  topics  for  each  of  the  two  systems  were  assigned  ranks 
based  on  scale  values  obtained  by  the  Scenario  Topics  Ranking  Survey. 


The  scenario  does  differ  depending  on  the  actions  taken  by  the  student. 
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Since  the  scenarios  were  the  same  for  the  two  trainers,  the  scale  values 
assigned  for  each  topic  should  have  been  approximately  the  same. 

Simply  stated,  the  ranks  for  the  two  systems  tracked  closely, 
except  for  communications,  which  was  ranked  the  eleventh  most  important 
for  the  IDCTT  Trainer  and  the  third  most  important  for  the  DC  Central 
Trainer.  Scale  values  for  each  topic  fell  within  one  point  for  each  of 
the  topic  between  the  two  methods  with  the  exception  of: 

•  Communications,  and 

•  Keeping  the  chain  of  command  informed, 

which  differed  by  only  2.05  and  1.14  points  respectively.  Figure  9 
displays  the  scale  values  obtained  for  the  two  trainer  types.  The  figure 
highlights  the  more  pronounced  emphasis,  as  determined  by  the  scale 
values,  for  the  topics  using  the  IDCTT  Trainer.  These  scale  values 
indicate  that  students  felt  the  degree  with  which  each  topic  was 

emphasized  was  more  pronounced  using  the  IDCTT  Trainer. 

(1)  Ranking  Differences.  As  shown  in  Figure  9, 

communications  and  keeping  the  chain  of  command  informed,  are  given 
different  priorities  in  the  two  systems.  The  content  of  student 
narratives  revealed  that  communications  were  not  emphasized  in  the  IDCTT 
Trainer  because  they  did  not  feel  the  touchscreen  input  device  effectively 
tested,  or  even  required,  their  communication  skills.  Students  felt  the 
DC  Central  Trainer's  group  environment,  combined  with  use  of  Sound  Powered 
Phone  Talkers  to  exchange  information,  emphasized  the  need  for  well 
practiced  communication  skills. 

Students  identified  problems  exchanging  information  with 
the  Bridge  and  the  Commanding  Officer  as  a  communication  deficiency  in  the 
IDCTT  Trainer.  This  shortcoming  may  have  induced  students  to  feel  that 
keeping  the  chain  of  command  informed  was  not  a  high  priority  in  the  IDCTT 
Trainer.  Conversely,  the  instructor  in  the  DC  Central  Trainer,  acting  as 
the  Bridge  and  the  Commanding  Officer,  continually  prompted  the  student 
for  information,  thus  creating  a  distinct  need  to  keep  the  chain  of 
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and  DC  Central  Trainers 

command  informed.  Although  the  data  did  not  indicate  which  method  was 
more  variable,  it  did  demonstrate  that  each  method  emphasized  the  same 

damage  control  fundamentals  with  the  exception  of  communications. 

b.  Interquartile  Range  Assessment 

Overall,  response  variability  for  the  IDCTT  Trainer  was 
higher  than  the  DC  Central  Trainer.  As  shown  in  Figure  10,  nine  out  of 
the  13  topics  (69  percent)  had  a  higher  IQR  value  for  the  IDCTT  Trainer. 
Students '  assessment  of  bow  important  each  topic  was  in  completing  the 
scenario  varied  more  for  the  IDCTT  Trainer.  If,  in  fact,  grading  the  13 
damage  control  topics  is  a  valid  measure  of  effectiveness  for  evaluating 
system  variability,  this  would  indicate  that  the  scenario  produced  using 
the  DC  Central  Trainer  is  less  variable  than  the  IDCTT  Trainer.  This 
finding  is  contrary  to  the  logical  expectation  that  the  computer 
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simulation  is  more  suitable,  by  design,  for  providing  identical  training 
scenarios.  If  the  IDCTT  Trainer  actually  provides  more  consistent 
scenarios,  then  either  the  measures  of  effectiveness  for  evaluating  system 
variation  are  incorrect  or  the  testing  methods  for  soliciting  students 
responses  were  unclear  or  not  the  best  method  possible.  Further  data  and 
analysis  of  the  systems'  variation  in  producing  consistent  training 
scenarios  is  required. 


Scenario  Topics  Banking* 

Survey  IQR  Results 

Low 

High 

IQR 

IQR 

0  12  3  4 

1  1  1  1  1 

5 

1 

fnmm  i  ifikldODS 

/  J  1 

Inform  Chain  of  Command 

(  x  ▼  -IDCTT 

Set  Zebra 

DC  Central- \  / - 

Trainer 

Restore  Vital  Systems 

Manage  DC  Central 

Isolate  Damage 

Manned  and  Ready  Reports 

Restore  Fbemain 

Manage  Personnel  Casualties 

Damage 

Prioritize  Damage 

Coordinate  Firefighting  Teams 

Isolate  Explosive  Hazards 

'  ^ 

Figure  10:  Fundamental  Damage  Control  Topics  Interquartile  Ranges  for 
the  IDCTT  and  DC  Central  Trainers 


C .  Summary 

The  data  collection  plan  provided  sufficient  information  to  conduct 
the  validation  analysis.  An  analysis  of  collected  data  indicated  that  the 
IDCTT  Trainer  is  an  effective  training  tool,  however  there  are  areas  where 
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improvements  can  be  made.  The  following  chapter  summarizes  the  findings 
of  this  validation  study. 
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V.  CONCLUSIONS 


The  findings  of  this  validation  indicate  that  the  IDCTT  Trainer  is  a 
highly  effective  training  aid  in  a  shore  based  environment.  Students  not 
only  reported  that  felt  the  IDCTT  Trainer  was  easy  to  operate  and 
extremely  useful  as  a  training  aid  at  the  Damage  Control  School,  they 
indicated  they  would  like  to  see  the  trainer  made  available  for  shipboard 
use.  When  compared  to  the  DC  Central  Trainer  on  specific  design  features, 
respondents  indicated  a  clear  preference  for  the  IDCTT  Trainer  on  every 
dimension  examined.  Further,  the  distribution  of  actual  performance 
scores  taken  from  the  two  different  trainers  revealed  that  the  IDCTT 
Trainer  is  clearly  the  more  effective  training  medium.  The  following 
sections  encapsulate  this  study's  major  findings  from  both  the  IDCTT 
Trainer  performance  evaluation  and  the  IDCTT  versus  DC  Central  Trainer 
performance  comparison. 

A.  IDCTT  TRAINER  PERFORMANCE  EVALUATION 

Taken  together,  the  data  from  the  IDCTT  Trainer  performance  evaluation 
revealed  that  the  system  was  enthusiastically  accepted  by  the  student  test 
group.  While  instructors  themselves  were  neutral  in  their  impression  of 
the  IDCTT  Trainer,  they  indicated  that  their  students  reacted  positively 
to  the  system.  The  results  from  this  circumscribed  portion  of  the  study 
were  presented  in  two  broad  categories;  IDCTT  Trainer  performance 
attributes,  and  suggested  areas  of  improvement.  The  findings  from  these 

topics  are  summarized. 

1.  IDCTT  Trainer  Performance  Attributes 

Students  indicated  that  the  IDCTT  Trainer  was  an  extremely 
effective  training  aid  and  the  IDCTT  Trainer  performance  evaluation 
specified  the  following  desirable  attributes. 

•  Students  liked  the  IDCTT  Trainer  concept  and  design. 

•  The  IDCTT  Trainer  was  easy  to  operate. 


•  The  trainer  was  consistently  considered  useful  as  a  DC  School 
training  aid  or  as  a  potential  shipboard  training  aid  across  all 
students  surveyed. 

•  Student  responses  were  clearly  favorable  to  the  IDcTT  on  each  of  the 
eight  conventional  interactive  courseware  dimension  presented. 

•  The  scenario  was  easy  to  understand  and  was  extremely  realistic. 

•  Audio  and  visual  effects  created  by  the  trainer  were  identified  as 

the  main  reasons  w'hy  the  trainer  was  so  successful  in  creating  what 

the  students  considered  to  be  a  realistic  training  scenario. 

•  The  students'  prevailing  experiences  that  the  scenario  was  difficult 
reflected  the  designers  original  intentions:  the  exercise  was  to  be 
difficult  . 

•  Scenario  objectives  were  clearly  defined. 

•  Respondents  rated  the  system  as  mentally  and  temporally  demanding 
while  requiring  a  high  level  of  effort  to  complete  the  IDCTT  Trainer 
scenario . 

•  Instructors  acknowledged  the  students'  positive  acceptance  of  the 
system . 

•  Students  quickly  became  familiar  with  the  IDCTT  Trainer. 

2 .  Improvement  Items 

Although  the  IDCTT  Trainer  was  agreed  upon  by  the  students  to  be 
an  effective  training  medium,  various  characteristics  of  the  trainer  were 
identified  for  potential  improvements.  These  areas  identified  by  the 
validation  study  are  listed  below. 

•  Most  of  the  problems  identified  with  the  system  were  rooted  in 
inputing  information  into  the  system  using  the  touchscreen  monitor. 

•  Fifty-nine  percent  of  the  students  indicated  they  experienced  some 
difficulty  operating  the  touchscreen  monitor. 

•  The  touchscreen's  primary  problem  was  its  low  sensitivity  to  touch 
and  its  slow  response  time. 

•  An  accidental  input  into  the  touchscreen  monitor  could  seriously 
affect  the  student's  performance  by  causing  them  to  focus  their 
attention  on  repeating  the  button  process  rather  than  the  flow  of 
information  from  the  various  sources. 

•  Students  recommended  increasing  the  sensitivity  of  the  touchscreen  or 
exploring  different  methods  for  inputing  information  into  the  system 
such  as  voice  activation. 

•  Thirty-eight  percent  of  the  students  indicated  they  had  problems  with 
the  pace  of  information  flow.  However,  most  students  appeared  to  be 
comfortable  with  the  pace  by  the  time  they  reached  their  graded 
sessior. . 


•  Students  and  Instructors  indicated  the  need  for  more  scenarios  with 
different  ship  classes  to  support  the  diversity  of  fleet  experience. 

•  The  main  problem  identified  with  the  Firemain  Panel  was  the  small 
display  screen. 

•  Students  often  overlooked  critical  indications  on  the  Firemain  Panel 
because  they  simply  did  not  notice  any  changes  in  its  compact 
disp  j.ay . 

•  Students  suggested  a  demonstration  tape  to  provide  a  step  by  step 
sequence  through  the  various  options  in  order  for  the  students  to 
familiarize  themselves  with  the  system  more  quickly. 

•  For  audio  reports,  respondents  indicated  that  they  became  distracted 
by  erroneous  information  in  the  background  audio  track. 

•  Fourteen  of  the  32  students  did  not  complete  the  scenario  due  to 
chill  water  related  problems. 

•  The  Damage  Control  School  should  review  its  course  curriculum  and 
place  a  stronger  emphasis  on  the  chill  water  system  if  the  curriculum 
is  to  reflect  the  shift  in  fleet  doctrine  toward  the  TSS  philosophy. 

•  Some  instructors  were  concerned  that  the  IDCTT  Trainer  lacked  a  clear 
mission  in  the  school's  curriculum. 


B.  IDCTT  VERSUS  DC  CENTRAL  TRAINER  PERFORMANCE  COMPARISON 

The  IDCTT  versus  DC  Central  Trainer  performance  comparison  was  used 
to  measure  the  IDCTT  Trainer's  performance  against  the  DC  Central  Trainer. 
This  portion  of  the  study  revealed  that  students  felt  that  the  IDCTT 
Trainer  was  the  preferred  training  method  in  every  category  examined. 
Specifically,  the  following  keynotes  were  identified. 

•  The  IDCTT  Trainer  is  by  far  the  preferred  training  method. 

•  The  IDCTT  Trainer  promoted  greater  learning,  produced  significantly 
more  stress,  and  stimulated  Students  to  learn  much  more  than  the  DC 
Central  Trainer  did. 

•  The  frequency  distribution  of  scores  received  using  the  IDCTT  Trainer 
were  higher  than  those  scores  received  using  the  DC  Central  Trainer 
at  the  .025  significance  level. 

•  Based  on  student  performance,  the  IDCTT  Trainer  is  a  more  effective 
training  medium. 

•  '/Then  students  were  exposed  to  the  stressful  IDCTT  scenario,  they 
appeared  to  concentrate  more  intently,  and  try  harder  than  during  the 
DC  Central  Trainer  sessions. 

•  Students  assumed  greater  responsibility  in  solving  the  scenario  and 
decision  making  became  more  spontaneous  while  exercising  in  the  IDCTT 
training  environment. 


Students'  assessment  of  how  important  each  fundamental  damage  control 
topic  was  in  completing  the  scenario  varied  more  for  the  IDCTT 
Trainer,  indicating  less  standardized  scenarios  than  the  DC  Central 
Trainer . 

Instructors  felt  that  the  IDCTT  Trainer  better  prepared  the  DCA 
students  for  the  actual  damage  control  situations  they  will  face  in 
the  fleet. 
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APPENDIX  A  SURVEYS 


The  following  surveys  were  the  surveys  used  in  support  of  the  data 
collection  plan.  All  seven  surveys  and  grade  sheets  which  were  used  in 
the  study  are  included  in  this  Appendix. 
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STUDENT  IDCTT  SURVEY 


NA>E  _  DATE _ 

RANK  _  YEARS  OP  SERVICE _ 

YEARS  ENLISTED  SERVICE _  PRIOR  ENLISTED  RATE _ 

WHAT  SHIPS  HAVE  YOU  SERVED  ON?  _  HOW  LONG 


Have  you  served  as  a  Damage  Control  Assistant  for  6  months  or  more  (CIRCLE 
ONE) ?  Yes  No 

Have  you  served  as  repair  2,  3  OR  5  Locker  Officer  for  6  months  or  more 
(CIRCLE  ONE)?  Yes  No 


Your  answers  to  the  following  questions  will  help  improve  the  quality 
of  training  you  receive  in  the  Damage  Control  Central  Simulators.  Please 

answer  the  following  questions  completely,  explaining  your  answers 
thoroughly.  Use  the  Back  of  the  questionnaire  if  additional  answer  space 
is  required.  Upon  completion,  please  return  this  survey  to  your 
instructor . 


1.  Approximately  how  much  time  did  you  spend  using  IDCTT? 
_  hours 


2.  Have  you  used  interactive  video  courseware  such  as  the  IDCTT  system 

before?  (CIRCLE  ONE)  Yes  No 

If  yes,  what  courseware  did  you  use?  (use  back  if  necessary) 

3 .  Rate  how  difficult  or  easy  the  IDCTT  System  was  to  operate? 

Very  Very 

Difficult  Neutral  Easy 

123456789  10  11 


4.  Check  any  of  the  following  operations  which  caused  you  difficulty  while 
operating  the  IDCTT  System. 

_  Inputing  information  with  the  touchscreen  monitor 
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Understanding  audio  reports 


Finding  DC  plate  information  with  ISMS  (examples:  compartment  and 
valve  numbers) 

Speed  ;r  volume  of  information  presented  (did  you  easily  lose  track 
of  the  situation  due  to  the  speed  or  volume  of  information  flow) 

Damage  control  alarm  panel  display 

Firemain  panel  and  firemain  valve  and  pump  options 
Other  (please  specify): 


5.  In  the  space  provided  below,  briefly  explain  why  the  items  you  checked 
caused  you  difficulty. 


6 .  Rate  the  extent  to  which  the  touchscreen  allowed  you  to  input  the 
information  necessary  to  combat  the  Damage  Control  scenario. 

All  of  the  None  of  the 

Information  Neutral  information 

123456789  10  11 


7 .  Rate  how  easily  the  touchscreen  allowed  you  to  input  information 

Very  Very 

Difficult  Neutral  Easily 

123456789  10  11 

8.  How  can  the  touchscreen  control  panel  be  improved? 


9.  Rate  the  IDCTT  Scenario  according  to  the  following  criteria  (note:  this 
question  refers  to  the  battle  problem  itself  and  not  the  IDCTT  system  as 
a  whole ) . 


Too 

Easy 

1 

2 

3 

4 

5 

Neutral 

6 

7 

8 

9 

10 

Too 

Difficult 

11 

Very 

Confusing 

1  2 

: > 

4 

5 

Neutral 

6 

7 

3 

9 

Very  Easy 
to  Understand 

10  11 

Too 

Fast 

1 

2 

4 

5 

Neutral 

6 

7 

8 

9 

10 

Too 

Slow 

11 

Very 

Realistic 

1  2 

4 

5 

Neutral 

6 

7 

8 

9 

10 

Very 

Unrealistic 

11 
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10.  What  problems  did  you  encounter  while  using  the  IDCTT  trainer? 


11.  What  aspects  of  the  IDCTT  Trainer  do  you  like  the  most? 


12.  What  aspects  of  the  IDCTT  Trainer  do  you  like  the  least? 


13.  Rate  how  useful  the  IDCTT  Trainer  is  as  simulation  training  aid  for 
students  at  the  Damage  Control  School? 


Very 

Useful  Neutral 

123456789 


Completely 

Worthless 

10  11 


14 .  Rate  how  beneficial  the  IDCTT  Trainer  would  be  as  an  installed 
shipboard  training  aid. 


Very 

Beneficial 

12  3  4 


Neutral 

5  6  7  8  9 


Completely 

Detrimental 

10  11 
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SUBJECT  INSTRUCTIONS  AND  SURVEY: 
SOURCES -OF  WORKLOAD  EVALUATION 


Throughout  this  experiment  the  rating  scales  are  used  to  assess  you 
experiences  in  the  different  tasks  conditions.  Scales  of  this  sort  are 
extremely  useful,  but  their  utility  suffers  from  the  tendency  people  have 
to  interpret  them  in  individual  ways.  For  example,  some  people  feel  that 
mental  or  temporal  demands  are  the  essential  aspects  of  workload 
regardless  of  the  effort  they  expended  or  the  performance  they  achieved. 
Others  fell  that  if  they  performed  well  the  workload  must  have  been  low 
and  vice  versa.  Yet  others  feel  that  effort  or  feelings  of  frustration 
are  the  most  important  factors  in  workload;  and  so  on.  The  results  of 
previous  studies  have  already  found  every  conceivable  pattern  of  values. 
In  addition,  the  factors  that  create  levels  of  workload  differs  depending 
on  the  tasks.  For  example,  some  tasks  might  be  difficult  because  the  must 
be  completed  very  quickly.  Others  may  seem  easy  or  hard  because  of  the 
intensity  or  mental  or  physical  effort  required.  Yet  others  feel 
difficult  because  they  cannot  perform  well,  no  matter  how  much  effort  is 
expended . 

The  evaluation  you  are  about  to  perform  is  a  technique  that  has  been 
developed  by  NASA  to  assess  the  relative  importance  of  six  factors  in 
determining  how  much  workload  you  experienced.  The  procedure  is  simple: 
Read  the  following  task  descriptions  and  then  mark  the  scale  at  the  point 
that  reflects  the  task  load  that  you  experienced.  If  you  have  any 
questions,  please  ask  them  now.  Thank  you  for  you  participation. 

Title  Endpoints  Descriptions 

Mental  Demand  Low/High  How  much  mental  and  perceptual  activity  was 

required  (e.g.,  thinking,  deciding, 
calculating,  remembering,  looking  searching, 
etc.)?  Was  the  task  easy  or  demanding,  simple 
or  complex,  exacting  or  forgiving? 

Physical  Demand  Low/High  How  much  physical  activity  was  required  (e.g., 

pushing,  pulling,  turning,  controlling, 
activating,  etc.)?  Was  the  task  easy  or 

demanding,  slow  or  brisk,  slack  or  strenuous, 
restful  or  laborious 

How  much  time  pressure  did  you  feel  due  to  the 
rate  or  pace  at  which  the  tasks  or  task 
elements  occurred?  Was  the  pace  slow  and 
leisurely  or  rapid  and  frantic? 

How  hard  did  you  have  to  work  (mentally  and 
physically)  to  accomplish  your  level  of 
performance? 

How  successful  do  you  think  you  were  in 
accomplishing  the  goals  of  the  task  set  by  the 
experimenter  (or  yourself)?  How  satisfied  were 
you  with  your  performance  in  accomplishing 
these  goals? 

How  insecure,  discouraged,  irritated,  stressed 
and  annoyed  versus  secure,  gratified,  content, 
relaxed  and  complacent  did  you  feel  during  the 
task? 


Temporal  Demand  Low/High 


Effort  Low/High 


Performance  Low/High 


Frustration  Low/High 


MENTAL  DEMAND 


How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking, 
deciding,  calculating,  remembering,  looking  searching,  etc.)?  Was  the 
task  easy  or  demanding,  simple  or  complex,  exacting  or  forgiving? 

Low  High 

123456789  10  11 


PHYSICAL  DEMAND 

How  much  physical  activity  was  required  (e.g.,  pushing,  pulling,  turning, 
controlling,  activating,  etc.)?  Was  the  task  easy  or  demanding,  slow  or 
brisk,  slack  or  strenuous,  restful  or  laborious? 

Low  High 

123456789  10  11 


TEMPORAL  DEMAND 

How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the 
tasks  or  task  elements  occurred?  Was  the  pace  slow  and  leisurely  or  rapid 
and  frantic? 

Low  High 

123456789  10  11 


PERFORMANCE  DEMAND 

How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the 
task  set  by  the  experimenter  (or  yourself)?  How  satisfied  were  you  with 
your  performance  in  accomplishing  these  goals? 

Low  High 

123456789  10  11 


EFFORT  DEMAND 

How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your 
level  of  performance? 

Low  High 

123456789  10  11 


FRUSTRATION 

How  insecure,  discouraged,  irritated,  stressed  and  annoyed  versus  secure, 
gratified,  content,  relaxed  and  complacent  did  you  feel  during  the  task? 

Low  High 

123456789  10  11 
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USER  INTERFACE  DIMENSIONS 


Directions:  A  number  of  statements  which  describe  the  Interactive-video 
Courseware  (ICW)  are  given  below.  Read  each  statement  and  then  make 
circle  the  number  which  reflects  your  opinion.  There  are  no  right  or 
wrong  answers . 


Dimension  1  -  Ease  of  Use 

(perceived  facility  with  which  user  interacts  with  the  ICW) 

Difficult  Easy 

123456739  10  11 


Dimension  2  -  Navigation 

(perceived  ability  to  move  through  the  contents  of  the  ICW) 

Difficult  Easy 

123456789  10  11 


Dimension  3  -  Cognitive  Load 

(Perceived  degree  that  the  user  interface  seems  manageable) 

Unmanageable  Manageable 

123456789  10  11 


Dimension  4  -  Mapping 

(Program's  ability  to  track  and  graphically  represent  user's  path  through 
the  program) 

None  Powerful 

123456789  10  11 


Dimension  5  -  Knowledge  Space  Compatibility 

(Network  of  concepts  and  relationships  that  compose  the  user's  knowledge 
about  the  topic) 

Incompatible  Compatible 

123456789  10  11 


Dimension  6  -  Information  Presentation 

(Perceived  degree  that  the  information  contained  in  the  ICW  is  presented 
in  an  understandable  form) 

Obtuse  Clear 

123456789  10  11 

Dimension  7  -  Media  Integration 

(How  much  does  the  ICW  coordinate  the  different  media  to  produce  an 
effective  whole) 

Uncoordinated  Coordinated 

123456789  10  11 
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Dimension  8  -  Overall  Functionality 

(Perceived  utility  of  the  ICW  in  relation  to  the  program's  intended  use) 

Dysfunctional  Highly  functional 

123456739  10  11 


IDCTT  INSTRUCTOR  SURVEY 


NAME 

DATE 

RANK 

YEARS  OF  SERVICE 

TIME 

AS 

AN 

INSTRUCTOR 

AT 

DCA  SCHOOL 

Your  response  to  the  following  items  will  help  modify 
the  IDCTT  program  to  more  specifically  address  your  Command's 
training  goals.  This  survey  is  designed  to  assess  how  instructors  rate 
different  aspects  of  the  IDCTT  Trainer  effectiveness.  There  are  also 
short  answer  questions  where  you  can  express  your  opinion  on  the  system. 
Please  answer  all  of  the  questions  completely,  explaining  your  answers 
thoroughly.  Use  the  back  of  the  questionnaire  if  additional  answer  space 
is  necessary. 


1.  Approximately  how  many  hours  did  you  spend  assisting  students  with  the 
IDCTT  Trainer? 

_  Hours 

2.  From  an  instructor's  perspective,  rate  how  easily  the  IDCTT  Trainer 
allowed  you  to  instruct  students  in  damage  control  simulator  training. 

Very  Very 

Easy  Neutral  Difficult 

123456789  10  11 

3 .  Rate  how  realistic  the  IDCTT  Trainer  depicts  damage  control  training 
compared  to  actual  shipboard  damage  control. 

Very  Very 

Realistic  Neutral  Artificial 

123456789  10  11 

4.  Rate  the  extent  to  which  you  would  like  to  see  the  IDCTT  Trainer  used 
as  a  permanent  aid  for  simulation  training  at  the  DC  School. 

Very  Very 

Much  Neutral  Little 

123456789  10  11 


5.  Rate  the  students  reaction  (positive  or  negative)  to  the  IDCTT  Trainer 
as  an  instructional  aid. 


Very 

Positive  Neutral 

1  2  3  4  5  6  7 


Very 

Negative . 

8  9  10  11 


6.  Rate  how  beneficial  the  IDCTT  Trainer  would  be  as  an  installed 
shipboard  training  tool. 


Very 

Beneficial 

12  3 


Neutral 

4  5  6  7  8  9 


Very 

Detrimental 

10  11 
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7 .  What  aspects  did  you  like  about  the  IDCTT  for  teaching  damage  control 
problems  ? 


8.  What  problems  did  you  encounter  while  using  the  IDCTT  as  an 
instructional  aid? 


9.  What  aspects  of  the  IDCTT  would  you  like  to  see  changed? 


10.  What  benefits  do  you  envision  from  the  use  of  IDCTT  at  the  Damage 
Control  School? 


-4 


STUDENT  GRADE  SHEET  FOR  IDCTT  AND  DC  CENTRAL 


TRAINER  SCENARIOS 


STUDENT'S  NAME  <DCA) _ DATE 

GROUP  MEMBER  NAMES  _ 


INSTRUCTOR'S  NAME  _ 

METHOD  (CIRCLE  (WE)  IDCTT  DC  CENTRAL  TRAINER 


The  following  is  a  grade  sheet  for  IDCTT  and  DC  Central  Trainers.  The 
results  will  be  used  ter  research  purposes  only:  Grades  assigned  using 
this  eva .  .at  1  :r.  form  will  Dot  effect  itadmt  axmdm*  la  «gj[  "«|wcltY.  This 
p er  f  ermanoe  evaluation  will  be  used  for  the  sole  purpose  of  comparing  the 
:r:TT  and  C .  lentra.  Trainer  methods. 

Maximum  points  allowed  for  each  topic  are  indicated  m  parentheses 
next  to  each  topics.  Partial  credit  may  be  awarded  when  actions  are  taker, 
for  a  top.c  area  but  r.ct  completely  or  correctly  executed.  For  example,  if 
fire  boundaries  were  successfully  set  on  5  of  5  spaces  requiring  this 
action,  the  student  would  receive  €■  cut  of  a  possible  10  points.  The 
instructor  will  determine  what  percentage  of  the  topic  area  was 
accomplished  and  assign  pcir.ts  accordingly. 


1.  DCA  ensured  zebra  was  set  or.  firemair.  _ 5  points 

2.  Investigators  sent  out  to  investigate  _  S  points’ 

ruptured  piping  when  pressure  indications 

suggested  a  rupture. 

i.  Maintained  firemain  pressure  through  _  (5  points ■ 

isolating  damaged  spots  and  firepump 
management . 

4.  Investigators  sent  out  to  investigate  _  5  p>oints 

all  fire  and  flooding  alarms. 

Fire  boundaries  set  around  compartments  _  11  points 

on  fire. 

-  .  Smoke  tour,  da:  les  set  around  compartments _ 5  points 

or.  fire. 


Flooding  boundaries  set  around  flooding  _ _ _  1  '  points 

f 1 coded  compartment s . 


rini.ty  focused  ;n  flies  fit  d?  near 

key  :-pa;~s  .  magazines .  i  ada:  equipment 


pair. 


Efforts  made  to  fight  fires  with  assets  _  (5  points) 

available . 

Efforts  made  to  isolate  ruptures  _  (5  points > 

non  firemair.  i.e.,  chill  water,  fuel, 
luce  oi. 


At'empts  made  *  :  repair  damage  when  _  S  points 

identified  i.e,  ruptured  chill  water, 

:  act  -red  firemair.  . 

•  he:  pr*rptei  for  acticr.  did  the  UCA  _  5  prints 

resp't.d  . an  effective  ana  timely 
manner  from  repan  letters,  or i dge .  C I C 

r.  :  '  prr.rp's  from  "SMC  cn  loss  of _ 1.  points 

n.l.  •4,»r  :  r  rverr.eat  tr.g  arrays  radars 

Mam*  air.  *  re  rig  picture  t  f  the  _ 11  points 

sar.a  ge  ,\r.  i  tire  urns  t  an  res  t  hat  he  wa  s 
-  rrr  a*  i  r.g 

Keep  *  r  -  rsiaar.ding  Cfficer  bridge  _  1  points 

.r.frrmed  r.  t  r.e  status  cf  damage  control 


TOTAL  POZirrs  SCORED 


ICC  point 


STUDENT  IDCTT  VS  DC  CENTRAL  TRAINER  COMPARISON 

SURVEY 


HAMS _  _  DATS. 

RANK  _ 


i  jr.sv»!  s  t;  the  fci  lowing  questions  will  help  improve  the  training 
.  :  #  ,-eive  ;r.  Damage  Control  Central  simulators.  This  survey  will 
t:  i*  for  comparing  the  effectiveness  of  the  IDCTT  Trainer 

ir. .  er.t  t  a  1  Trainer  as  a  training  aids.  This  survey  is  designed  to 
ie*  -:n:.e  /cur  opinion  cn  the  ability  of  the  IDCTT  and  DC  Central  Trainers 
p:  quality  training.  If  you  haea  any  coaiBts  that  the  numerical 

scale  do  not  address,  plaasa  writ#  your  coanenta  on  the  bark  of  tha 
survey  "p  r.  completion.  please  return  this  form  to  your  instructor. 


rate  which  method  provides  a  more  reai.s  model  of  an  actual 

shipboard  damage  control  environment. 

IDCTT  Nmutrml  DCC  Trminmr 

:  -  >  4  5  6  -  s  ?  10  u 

rate  which  method  provides  the  ability  for  the  instructor  to  provide 
*  he  most  complete  post  scenario  debriefs. 

IDCTT  Nmutrml  DCC  Trainer 

1  2  2  4  5  6  7  8  9  10  11 

i .  Rato  which  method  produced  the  greatest  level  of  stress  while 
performing  the  damage  control  scenario. 

IDCTT  Neutral  DCC  Trminmr 

123456789  10  11 

4 .  Rate  which  system  enabled  the  instructors  to  monitor  student 
performance  more  closely. 

IDCTT  Nmutrml  DCC  Trminmr 

123456789  10  11 

5.  Rate  which  method  will  better  prepare  you  for  the  actual  casualties 
that  might  encounter  while  onboard  a  ship. 

XJDCTT  Neutral  DCC  Trminmr 

123456789  10  11 


6.  Rate  which  method  provides  an  easier  means  to  take  action  on  scenario 
problems  (i.e.,  sound  powered  phones,  DC  plate  plotting  and  hardwired 
alarm  panels  vise  touchscreen  inputs  and  computer  monitor  alarm  panels) 

IDCTT  Neutral  DCC  Trainer 

123456789  10  11 
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7.  Rate  which  method  provides  scenario  information  in  a  manner  most 
closely  resembling  shipboard  emergency  situations. 

IDCTT  Neutral  DCC  Trainer 

123456789  10  11 

8.  Rate  which  method  is  more  effective  in  teaching  damage  control  skills 
necessary  to  combat  damage  control  problems 

IDCTT  Neutral  DCC  Trainer 

123456789  10  11 

9 .  Rate  which  system  promoted  greater  learning  in  the  amount  of  time 
allocated. 

IDCTT  Neutral  DCC  Trainer 

123456789  10  11 

10.  If  you  had  access  to  one  method  of  instruction  rate  which  system  you 
would  prefer. 

IDCTT  Neutral  DCC  Trainer 

123456789  10  11 


11.  Rate  which  method  inspires  you  to  perform  to  the  best  of  your  ability. 

IDCTT  Neutral  DCC  Trainer 

123456789  10  11 
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SCENARIO  TOPICS  RANKING 


NAME _  DATE _ 

THIS  RANKING  IS  FOR  (CIRCLE  ONE) :  IDCTT  SCENARIO 

DC  CENTRAL  TRAINER  SCENARIO 


The  following  thirteen  topics  summarize  the  principal  actions 
necessary  to  successfully  complete  a  damage  control  problem.  The 
instruction  method  to  which  you  were  just  exposed  emphasized  each  of  the 
following  topics  to  varying  degrees.  In  the  blank  space  provided  to  the 
left  of  each  topic,  rank  each  topic  from  1  to  7  based  on  how  you  felt 
the  method  (IDCTT  or  DC  Central  Trainer)  emphasized  the  importance  of  aach 
topic  to  complete  the  DC  problem.  Use  the  following  criteria  to  express 
your  opinion: 

Not  At  All  Sxtrmaaly 

Important  Nautral  Important 

1  2  3  4  5  6  7 

NOTE:  Base  your  rankings  on  what  the  scenario  emphasized  and  not  on  what 
you  think  should  be  emphasized. 


Maintain  effective  communications 
Keep  the  chain  of  command  informed 
Ensure  Zebra  is  set 

Restore  vital  systems  (i.e.,  mechanical,  chill  water,  ventilation) 
Manage  Damage  Control  Central 

Isolate  damage  (smoke,  fire  and  flood  boundaries) 

Confirm  proper  manned  and  ready  reports 
Restore  Firemain 

Manage  personnel  casualties  and  evacuations 
Locate  damage 
Prioritize  casualties 

Coordinate  firefighting  and  repair  teams 

Isolate  explosive  hazards  (i.e.,  magazines,  fuel  tanks) 


3? 


APPENDIX  B  STUDENT  AND  INSTRUCTOR  ESSAY  RESPONSES 


IDCTT  Trainer  Survey  Question  5 

Briefly  explain  why  any  of  the  below  listed  IDCTT  Trainer  features  caused 
you  difficulty  while  operating  the  system. 

A.  Inputmg  information  with  the  touchscreen  monitor 

Trying  to  input  information  rapidly  with  large  fingers  on  a  relatively 
small  screen. 

More  distance  needed  between  touchscreen  buttons  to  avoid  accidental 
inputs . 

Touchscreen  needs  to  be  more  sensitive. 

Touchscreen  did  not  always  take  information  first  time  it  was  entered. 
Too  slow  when  ordering  actions. 

Touchscreen  responded  too  slowly. 

Touchscreen  looked  like  it  activated  only  to  find  out  that  it  did  not. 
Difficult  to  press  correct  buttons. 

Missed  information  while  typing  in  orders  on  the  touchscreen. 

Screen  did  not  respond  promptly  to  inputs. 

Had  difficultly  getting  to  appropriate  menu. 

Did  not  know  if  orders  were  accepted. 

Difficult  to  input  data. 

Difficult  inputmg  information. 

Did  not  take  information  inputs. 

Hard  to  get  used  to  touchscreen. 

Due  to  the  speed  of  the  scenario  it  was  difficult  to  input  necessary 
information  in  time  given. 
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Unfamiliarity  with  the  inputing  system  and  the  ship  involved  made  it 
difficult  to  enter  information. 

Touchscreen  was  not  very  sensitive,  I  usually  had  to  hit  the  screen  2 
times  to  activate  the  option. 

Sometimes  you  had  to  touch  the  screen  two  or  three  times  to  input 
information . 

Events  were  overwhelming  at  times  due  to  the  slow  data  input  rate. 

It  is  easy  to  get  overwhelmed  by  incoming  information  because  you  can 
not  give  immediate  responses  to  the  reports,  so  incoming  reports  mount 
up  before  the  DCA  can  tell  the  computer  what  to  do. 

Entering  data  was  difficult  due  to  unfamiliarity  with  the  sequencing 
logic  and  symbology.  It  got  easier  as  the  program  progressed. 

Some  options  were  difficult  to  discern  or  are  not  activated. 

B.  Understanding  audio  reports 

Background  noise  sounds  like  a  report  because  it  keeps  saying, 

“DCA . “. 

Background  noise  contained  clearly  erroneous  information. 

Could  not  ask  for  repeat  of  voice  messages. 

C.  Finding  DC  plate  information 

Never  had  seen  DDG-51  chill  water  or  compartment  DC  Plates,  which  led  to 
problems  isolating  systems. 

Not  used  to  DDG-51  Class. 

DC  Plates  difficult  to  understand. 

Could  not  find  the  valve  numbers  I  wanted  to  close. 

Could  not  locate  the  information  needed. 

Z.  Speed  or  volume  of  information  presented 

Too  fast  for  a  beginner,  need  to  be  able  to  adjust  speed  with  which 
information  is  given. 

Speed  of  information  presented  overloads  the  DCA  at  first. 

Too  many  printed  messages. 

Lost  track  cf  audio  reports  and  tried  to  key  on  important  issues. 

Massive  volume  of  data  was  difficult  to  track  and  remember  but  this 
seems  to  make  the  simulation  more  realistic. 
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E.  Damage  control  alarm  panel  display 
No  responses . 

F.  Firemain  panel  and  firemain  valve  and  pump  operations 
The  firemain  panel  is  busy  and  it  goes  by  fast. 

Firemain  screen  is  too  small,  can't  read  print. 

Firemain  panel,  especially,  fire  pumps  were  difficult  to  read  and 
understand. 


G.  Other 

The  initial  five  to  ten  minutes  of  the  first  run  was  just  system 
familiarization . 

Not  able  to  find  out  the  status  of  the  chill  water  system. 

Big  trouble  learning  what  actions  the  repair  lockers,  etc,  were  doing  on 
their  own  and  what  I  needed  to  direct.  Also,  what  I  had  to  tell  lockers 
to  get  things  done . 

Lack  of  experience  (my  problem) . 

Tried  to  order  a  COV  to  Repair  V  when  it  was  in  the  Repair  III  area. 
Computer  just  said,  “review  your  last  order"  and  did  not  say  that  the 
valve  was  not  in  their  area.  Locker  should  say,  "That  is  in  Repair  III 
area  Sir ! " . 

The  program  will  not  accept  spaces  to  investigate  unless  damage  is 
reported  in  the  space. 

Unable  to  give  space  name  to  locker  and  let  the  locker  look  up  the  fire 
and  flooding  boundaries. 


IDCTT  Trainer  Survey  Question  8 

How  can  the  touchscreen  control  panel  be  improved? 

I  think  it  is  fine. 

The  screen,  at  times,  does  not  respond  well. 

Larger  number  pad  on  touchscreen,  use  keyboard  instead. 
Put  more  space  in  between  buttons . 

Increase  touchscreen  sensitivity. 
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Better  touchscreen  sensitivity. 

Practice . 

Make  touchscreen  more  sensitive  to  the  touch  and  quicker  to  respond. 

Auto  advance  through  "setting  boundaries"  after  enter  is  pushed. 

Teach  the  users  the  proper  method  of  touching  the  screen. 

Program  a  five  to  seven  second  pause  into  the  system  to  input 
information.  Voice  activated  commands  would  be  best. 

More  sensitive  with  quicker  response  time. 

Faster  button  response  to  the  touch. 

Touchscreen  does  not  always  “take"  information  when  you  push  the  area, 
other  times  you  have  to  keep  your  finger  on  it  for  a  long  time  before  it 
accepts  your  input. 

Difficult  to  enter  information  on  screen,  improve  screen  sensitivity. 
Add  an  audio  bell  to  indicate  that  the  inputed  information  was  accepted. 
Use  mouse  control  to  increase  speed. 

Input  information  through  the  keyboard  only. 

Switch  to  a  mouse. 

Switch  to  voice  activation. 

Make  status  and  repair  keys  give  better  information. 

Use  keyboard  vice  screen  keypad. 

Not  all  actions  are  acknowledged  the  same  way  causing  multiple  inputs. 
Improve  system  so  screen  does  not  freeze  up. 

Improve  sensitivity. 

Voice  activation  of  DCA  orders. 


IDCTT  Trainer  Survey  Question  10 

What  problems  did  you  encounter  while  using  the  IDCTT  Trainer? 

The  Command  Console  number  pad  is  slow  in  responding  to  inputs. 
Unfamiliarity  with  DDG-51  systems. 

No  method  for  confirming  actions  ordered  were  taken.  While  inputing 
information  into  the  system,  I  accidentally  cancelled  the  input. 

Inputing  information  through  the  touchscreen  and  background  noise. 
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Dirty  screen  causing  the  computer  to  act  funny.  We  cleaned  the  screen 
and  everything  was  fine. 

Learning  what  functions  each  button  option  would  do. 

Losing  information  inputed  through  the  screen  because  I  did  not  press 
“OK". 

Initial  system  familiarization. 

System  froze  to  reload  printer  paper  and  some  kind  of  memory  error. 

Coordinating  all  reports  and  correlating  them  to  the  various  plots  and 
displays . 

No  way  of  asking  the  status  of  boundaries  being  set. 

Focusing  energy  on  inputing  information  rather  then  the  battle  problem. 

Unfamiliarity  with  the  system,  need  more  time  to  practice. 

Lost  control  of  the  situation,  typed  in  flooding  boundaries  when  I 
wanted  to  set  fire  boundaries. 

Lack  of  knowledge  of  chill  water  system  killed  me. 

Inexperienced  in  making  reports  to  the  commanding  officer. 

Inputing  information  into  the  touchscreen. 

Printer  stalled  during  the  battle  problem. 

System  would  lock-up  for  a  couple  of  seconds  for  unknown  reason. 

Message  blanks  are  good  but  too  much  information  is  given  in  the  amount 
of  time  allotted  to  read  them. 

No  clear  means  of  reporting  information  to  the  Captain/Bridge. 
Confusion  caused  by  background  noise  that  gave  incorrect  reports. 

Lack  of  experience  with  equipment . 

Finding  the  correct  keys  to  touch. 

Knowing  what  was  done  automatically.  For  example  during  beginning 
sequence,  zebra  on  Firemain  is  reported  set,  but  I  had  to  order  the 
valve  closed. 

Learning  what  to  press  to  make  reports  to  the  Commanding  Officer  was 
difficult . 

Lack  of  knowledge  of  how  to  use  it. 

Touchscreen  sensitivity  and  becoming  familiar  with  the  computer. 

Knowledge  of  the  repair  locker  capabilities  and  what  I  had  to  tell  them 
to  carry  out  what  I  wanted,  ex:  fight  for  fighting  fires. 

Sometimes  correct  inputs  were  not  understood  by  the  computer  (ie,  fight 
fires  C-l  level  was  accepted  but  actual  compartment  number  was  not) . 
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The  magazine  blew  up  even  though  I  lit  off  the  deluge  system. 

Lack  of  familiarity  with  the  system. 

Dirty  screen  caused  faulty  input  responses.  Unfamiliarity  with  command 
and  control  functions  caused  some  confusion  early  on  but  became  less  of 
a  problem  as  experience  increased. 


IDCTT  Trainer  Question  11 

What  aspects  of  the  IDCTT  Trainer  did  you  like  the  most? 

Trainer  seems  realistic. 

Interactive  aspects. 

Clarity  of  information  received. 

Realistic  flow  of  events  and  speed  of  damage .  DCA  gets  an  actual  test 
rather  than  knowing  what  will  happen. 

Amount  of  stress  induced  by  the  system. 

Good  system  for  training,  don't  have  to  go  to  GQ  to  operate. 

Audio  was  excellent  and  message  blanks  provided  added  realism. 

Ability  for  one  person  to  train  without  the  ship  going  to  GQ. 

Being  able  to  repeat  a  scenario  until  you  get  it  right. 

Audio  and  video  messages. 

Visually  realistic  representation  of  DC  member  images. 

Pace  and  presentation  of  information. 

Very  realistic. 

Clear  information  through  printer  messages. 

Printed  DC  messages. 

Video  images  of  people  telling  you  information. 

Fast  and  furious  flow  of  information  seems  fairly  realistic. 

Problems  cascade  if  appropriate  actions  are  not  taken. 

Fast  paced  audiovisual  displays  make  you  think  quickly. 

Repetition  of  the  same  scenario  allows  you  to  learn  the  proper  sequence 
of  actions  to  combat  the  damage. 

Good  training  aid  that  focuses  on  the  DCAs  skill. 

Visual  display  of  actions  taking  place. 
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Real  life  feeling. 

Speed  and  reality  of  scenario. 

Realistic,  fast  paced,  you  as  a  student  became  very  involved.  I  was 
impressed . 

Stresses  the  DCA  and  starts  preparing  him  for  how  he  needs  to  think  in 
DC  Central . 

Stress  technical  aspects  of  DC,  such  as  boundaries  and  reenforces  them 
much  better  then  the  DC  Central  Trainer. 

Video  display  makes  it  look  very  realistic. 

Sound  effects  and  voice  reports. 

Video  and  audio  effects. 

One  man  trainer  does  not  require  any  ship's  assets  to  initiate. 

High  intensity,  realistic  locker  actions,  no  stupid  locker  leaders. 

It  did  not  take  a  whole  watch  team  to  get  training.  Individual  training 
is  a  lot  easier  to  schedule  in  real  life. 

The  fast  paced  multiple  casualties. 

Lets  you  see  the  consequences  of  your  actions,  ie .  securing  chill  water 
makes  you  lose  you  Aegis  system. 

Very  realistic  and  stressful.  Seems  to  fairly  represent  an  actual 
damage  scenario.  It  forces  you  to  prioritize  casualties  an  allot 
resources  accordingly. 

The  background  noise,  the  pictures  of  personnel  talking  to  you,  the  many 
options,  this  all  made  it  very  realistic. 

IDCTT  Trainer  Survey  Question  12 

What  aspects  of  the  IDCTT  Trainer  did  you  like  the  least? 

Waste  of  paper  in  repeatedly  printing  out  DC  chits. 

Only  one  scenario. 

Pace  of  scenario. 

Difficulty  inputing  information. 

Difficulty  while  inputing  information. 

Firemain  screen  is  too  small. 

Comments  from  post  scenario  reasons  for  kill  point  activation  are  too 
general . 
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Trying  to  use  the  DC  Plates  while  operating  the  computer  was  difficult. 

Potential  for  other  damage  control  team  members,  who  do  not  have  this 
system,  to  go  untrained. 

Not  being  able  to  tell  someone  verbally  what  to  do. 

Not  all  options  are  clearly  expressed  in  the  Command  Console's  menu. 
Touchscreen . 

Not  able  to  order  three  things  at  once. 

Firemain  control  panel . 

Chill  water  seems  to  be  a  big  emphasis,  and  it  should  be,  but  some 
amount  of  ship  specific  knowledge  is  required  to  do  well  on  the 
simulator,  so  this  facet  is  pretty  pointless  unless  you  are  going  to  a 
DDG-51. 

Some  options  on  the  menu  could  not  be  initiated. 

Need  more  drill  scenarios. 

Need  mere  shxp  classes. 

Did  not  drill  the  entire  Damage  Control  Central  watch  team. 

Limited  options  available. 

Need  different  scenarios. 

Does  not  train  how  to  handle  DC  Central  watch  team. 

The  DCAs  load  is  more  because  he  has  to  type  information  into  the 
computer  while  in  the  DC  Central  Trainer  he  is  less  loaded  because  he 
talks  and  directs . 

Inaccurate  information  in  background  noise  is  distracting  and 
misleading . 

Method  of  issuing  orders. 

Definitely  a  game. 

DCA  does  not  sit  at  a  computer,  he  stares  at  charts.  If  the  DCA  could 
stand  and  plot  and  order  actions  it  would  be  better.  This  could  be 
solved  by  having  the  DCA  stand  and  have  a  computer  operator  input  his 
orders . 

Was  not  able  to  jumper  systems. 

Data  input  slow.  Touchscreen  keyboard  locked  up  a  few  times.  Sometimes 
hard  to  get  computer  to  do  what  you  want  it  to  do. 

It  takes  a  relatively  long  time  to  input  data.  Even  if  one  is  familiar 
with  keyboards,  data  input  is  slow  compared  to  voice  commands. 

Could  respond  more  quickly  in  a  real  locker.  Did  not  like  having  to 
input  every  small  detail,  but  it  did  make  you  think  things  out. 
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Only 


on#  scenario 


IDCTT  Instructor  Survey  Question  7 


Wr. a",  aspects  did  y like  about  tr.e  IDCTT  Trainer  for  teaching  damage 
centre!  problems; 

Em.phas.ced  tne  basic  sequence  of  damage  centre!  problems  evolution 
i.e..  set  boundaries,  mechanically  and  electrically  isolate,  fight 
f ires ,  etc .  . 

It  objectively  allows  for  multiple  paths  to  be  initiated,  which  is 
difficult  tc  replicate  m  the  DC  Central  Trainer. 

Ability  to  present  identical  scenarios  to  each  DCA  student. 

The  ability  to  objectively  critique  the  DCA's  actions  (i.e.,  prevents 
the  "No,  I  didn ' t “ . . . " Yes ,  you  did"  scenario). 

Printed  message  blanks. 

Realism  of  noise  and  confusion  that  would  occur  during  a  damage  control 
problem. 

The  realism,  in  the  fleet  it  is  very  hard  to  reproduce  the  realistic 
pressure  involved  in  DC  scenarios . 

The  stress  level  that  it  creates.. 

Imposes  very  realistic  level  of  stress  on  student,  particularly  through 
the  amount  of  informant  given,  the  manner  in  which  the  information  is 
given,  and  in  the  time  in  which  it  is  given. 

Reinforces  basic  and  advanced  damage  control  concepts  (some  with 
immediate  feedback,  some  with  delayed  feedback,  i.e.,  ending  program). 


IDCTT  Instructor  Survey  Question  8 

What  problems  did  you  encounter  while  using  the  IDCTT  as  an 
instructional  aid? 

The  program  did  not  run  in  real  time.  It  moved  more  like  a  video  game 
than  an  actual  scenario. 

Student  familiarity. 

Certain  situations  could  only  be  corrected  by  one,  and  only  one  action. 
Real  time  versus  simulator  time. 

Speed  of  input,  the  touchscreen  frustrated  students. 

Touchscreen  slowed  DCA's  responses. 

Students  unfamiliar  with  the  DDG-51  Class  platform. 
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pr«par • 


Fair  iv  extensive  pre-brief  ;n  riassr  ccit  required  t:  fully 
students  fez  IDC*’*?. 

IDCTT  interface  still  a  Pit  limiting  time  wise 

IDCTT  l::ks  up  rccas lonal ly  very  seldom  new  . 

Problems  with  inactive  buttons  on  touchscreen  monitor  ‘having  to  explain 
they  do  not  function 


IDCTT  Instructor  Survsy  Question  9 


/.'hat  aspects  of  the  IDCTT  Trainer  would  you  like  to  see  changed? 

CSMC  should  have  its  own  repair  team  that  isolates  and  corrects  chill 
water  problems.  The  ship  should  not  rely  heavily  on  the  DCA  for  Combat 
System  casualties. 

The  DCA  has  to  prompt  too  many  actions.  Fire  and  flooding  boundaries, 
and  isolation  would  be  reported  to  the  DCA  rather  than  ordered  by  the 
DCA. 

Increase  the  avenues  in  which  the  DCA  may  approach  solutions  to  the 
scenario . 

Provide  feedback  from  the  lockers  in  response  to  incorrect  or  improper 
orders . 

Change  touchscreen  to  voice  recognition. 

Obviously,  in  the  future,  IDCTT  will  ideally  have  multiple  scenarios  and 
multiple  platforms  to  choose  from. 

More  versatility.  The  IDCTT  Trainer  needs  to  have  more  scenarios  and  if 
the  DCA  could  program  IDCTT  himself,  it  would  add  more. 

Scenario  made  a  little  more  robust  by  adding  more  options  for  the 
students  to  initiate  various  orders. 

More  scenarios. 


IDCTT  Instructor  Survey  Question  10 


What  benefits  do  you  envision  from  the  use  of  the  IDCTT  Trainer  at  the 
Damage  Control  School? 

If  IDCTT  can  be  "connected"  to  an  EOOW,  OOD  and  CICWO  Trainer  to  do  an 
integrated  ship  training  evolution,  this  will  be  a  vehicle  for  us. 

May  be  used  as  a  final  simulation  to  objectively  determine  a  students' 
ability  to  operate  under  stressful  conditions. 
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The  ability  to  better  prepare  a  student  Cor  the  pressures  and  problems 
associated  with  liva  DC  scenarios  in  tha  fleet. 

It  tetter  prepares  DC  As  to  face  the  stresses  of  real  shipboard 
disaster  s 

Better  preparing  the  CCA  for  the  real  situation. 
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APPENDIX  C  WILCOXON  SIGNED-RANK  TEST  CALCULATIONS 


Null  Hypothesis:  H,:  The  population  distribution  for  the 

IDCTT  Trainer  and  DC  Central  Trainer  performance  scores  are 
ident ical . 

Alternative  Hypothesis:  Hd:  The  population  relative  frequency 
distribution  of  the  IDCTT  Trainer  performance  scores  is 
shifted  to  the  right  of  the  DC  Central  performance  scores. 


Data: 

T*  =  Rank  sum  of  the  positive  differences 
=  353 


T'  =  Rank  sum  of  the  negative  differences 

=  143 

n  -  31 

Note:  Table  contains  the  difference  and  rank  scores  for  32 
student  participants 

Test  Statistic:  z  =  1.99  (calculated  from  below  equation) 

2~  r--(n(n-q)/4i 

VMn*  1)  (2n-l)/24 


Rejection  Region:  Reject  H0  if  z  >  za 

z  =1.99 

z0  =  1.96  at  the  .025  significance  level 

Conclusion:  Reject  the  null  hypothesis  at  the  .025  significance 
level .  The  distribution  of  performance  scores  for  the  IDCTT 
Trainer  is  higher  then  the  distribution  of  performance  scores 
for  the  DC  Central  Trainer. 
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TABLE  17:  STUDENT  TRAINER  SCORES 


Student 

IDCTT 

DC 

Central 

Student 

IDCTT 

DC 

Central 

1 

94 

83 

17 

88 

91 

2 

72 

88 

18 

95 

93 

3 

87 

88 

19 

82 

83 

4 

83 

85 

20 

87 

79 

5 

88 

79 

21 

92 

81 

6 

91 

91 

22 

95 

81 

7 

96 

85 

23 

83 

93 

8 

87 

93 

24 

92 

85 

9 

92 

82 

25 

93 

82 

10 

92 

93 

26 

93 

91 

11 

90 

79 

27 

85 

91 

12 

89 

88 

28 

92 

93 

13 

77 

88 

29 

99 

88 

14 

93 

85 

30 

82 

81 

15 

81 

79 

31 

98 

85 

16 

97 

93 

32 

94 

93 

TABLE  18:  IDCTT  AND  DC  CENTRAL  TRAINER  WILCOXON  RANKINGS 


Student 

Difference 

Rank 

Student 

Difference 

Rank 

1 

11 

25 

17 

-3 

12 

2 

-16 

31 

18 

2 

9.5 

3 

-1 

4 

19 

-1 

4 

4 

-2 

9.5 

20 

8 

17.5 

5 

9 

19 

21 

11 

25 

6 

0 

None 

22 

14 

U> 

o 

7 

11 

25 

23 

-10 

20.5  I 

8 

-6 

14.5 

24 

7 

16 

9 

10 

20.5 

25 

11 

25 

10 

-1 

4 

26 

2 

9.5 

11 

11 

25 

27 

-6 

14.5 

12 

1 

4 

28 

-1 

4 

13 

-11 

25 

29 

11 

25 

14 

8 

17.5 

30 

1 

4 

15 

2 

9.5 

31 

13 

29 

16 

4 

13 

32 

1 

4 

103 


LIST  OF  KSFKltBICIS 


David  Taylor  Research  Center  Ship  Structure  and  Protection  Department, 
Total  Ship  Survivability  Fleet  Training  (TSS/FT)  Flooding  Model  User  Guide 
-  Version  1.0,  By  M.  Padgett  and  H.  Wolk,  p.2,  June  1992. 

Edwards,  A.L.,  Techniques  of  Attitude  Scale  Construction,  pp.  87-88, 
Appleton-Century-Crof ts ,  Inc.,  1957. 

Gritzen,  E.F.,  Introduction  to  Naval  Engineering,  pp.372,  Naval  Institute 
Press,  1980. 

Hart,  S.  G.,  and  Staveland,  L.E.,  Human  Mental  Workload,  Amsterdam: 
Elsevier,  1988 

Interview  between  D.  Jullian,  Lieutenant,  USN,  Surface  Warfare  officer 
School  Damage  Control  Department,  Newport  Rhode  Island,  and  the  author,  20 
April  1994. 

Mendenhall,  W.,  Scheaffer,  R.L.,  and  Wackerly,  D.D.,  Mathematical 
Statistics  With  Applications ,  p.680,  PWS-Kent  Publishing  Company,  1990. 

Naval  Education  and  Training  Command,  Surface  Warfare  Officer  Damage 
Control  Assistant  [A-4G-0020 )  Course  Curriculum  Outline,  p.4-20,  April 
1991). 

Naval  Sea  System  Command,  Integrated  Survivability  Management  System 
Training  Manual,  p.l,  1992. 

Reeves,  T.  C.,  and  Harmon,  S.  W.,  "Systematic  Evaluation  Procedures  for 
Instructional  Hypermedia/Multimedia, "  paper  presented  at  the  Annual 
Meeting  of  the  American  Education  Research  Association,  Atlanta,  Georgia, 

1993. 

Surface  Warfare  Officer  School  Command  Report,  Problem  Description  and 
Needs  Justification  for  Interactive  Damage  Control  Training  Module,  by 
D.R.  Monroe,  p.3,  4  February  1993. 

Ulozas,  B.,  "Surface  Warfare,  Enhancing  the  Capability  of  Damage  Control 
Assistants,"  paper  presented  ant  the  Chief  of  Naval  Operations  Deunage 
Control  and  Firefighting  Working  Group,  Hampton,  Virginia,  15  November 
1993. 

Ulozas,  B.,  "Total  Ship  Survivability,  Integrated  Damage  Control  Training 
Technology, "  paper  presented  at  the  Chief  of  Naval  Operations  Damage 
Control  and  Firefighting  Working  Group,  Hampton,  Virginia,  15  November 
1993  . 

United  States  Naval  Health  Sciences  Education  and  Training  Command, 
Interactive  Multimedia  Courseware  Validation  Report,  p.2,  3  December  1992. 


104 


INITIAL  DISTRIBUTION  LIST 


No. 

1.  Defense  Technical  Information  Center 

Cameron  Station 
Alexandria  VA  22304-6145 

Library,  Code  052 

Naval  Postgraduate  School 

Monterey  CA  93943-5002 

Commander  Frank  C.  Petho,  MSC,  USN 
Naval  Postgraduate  School  (OR/PE) 

Monterey,  California  93943-5000 

4.  Professor  Lyn  R.  Whitaker 

Naval  Postgraduate  School  (OR/WH) 

Monetery,  California  93943-5000 

5.  Lieutenant  Mark  S.  Johnson,  USN 
Department  Head  Class  135 
Surface  Warfare  Officer  School 
Newport,  Rhode  Island  02841-5012 

A.  Doctor  Bernard  J.  Ulozas,  D.A.,  Code  132 

Navy  Personnel  Research  and  Development  Center 
San  Diego,  California  92152-6800 

7.  Commander,  Naval  Sea  Systems  Command,  (NAVSEA-06K2 ) 

Atten:  Mark  McLean 

Washington,  D.C.  20362-5101 

8.  Commander  John  A.  Kunert,  USN 

Director,  Damage  Control  Training  Department 
Surface  Warfare  Officers  School  Command 
Newport,  Rhode  Island  02841-5012 

9.  Systems  Integration  and  Research,  Inc. 

Atten:  Donald  M.  Budai,  Captain,  USN  Ret. 

15412  Electronic  Lane 

Huntington  Beach,  California  92694 

10.  Doctor  Eric  B.  Allely,  M.D. 

Director,  USUHS-CIM 

4301  Jones  Bridge  Road 
Bethesda,  Maryland  20813-4799 


Copies 

2 


3 


1 


1 


1 


2 


1 


1 


1 


105 


